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Overview 


PACE Center for Girls is a unique program that provides academic and social services to girls ages 
11 to 18. Girls eligible for PACE exhibit multiple health, safety, and delinquency risk factors, such 
as poor academic performance, exposure to abuse or violence, truancy, risky sexual behavior, and 
substance abuse. PACE seeks to help them onto a better path and reduce the likelihood of negative 
outcomes, such as involvement with the juvenile justice system. 

PACE operates 19 nonresidential, year-round program centers across the state of Florida. Girls 
attend PACE daily during normal school hours and receive academic and extensive social services 
in a gender-responsive environment — that is, one tailored to the needs of girls. Most girls plan to 
attend PACE for approximately one year; during this time, they receive academic instruction and 
advising, a life skills curriculum, assessment and care planning, individual and group counseling, 
and service learning and work readiness opportunities. Parental engagement and transition and 
follow-up services are also key components of the PACE program. When girls leave PACE, they 
often return to other schools in their communities to complete their secondary education. 

This report presents implementation research findings from MDRC’s ongoing evaluation of 14 
PACE centers. A final report presenting the impacts of the program is planned for release in 20 1 8. 

Key Findings 

• The PACE program model, defined through both general program principles and a detailed 
manual, was implemented consistently across multiple locations. Girls at PACE received most 
services at the intended levels. Services varied somewhat across locations because of differences 
in staff and local resources and in program areas where the model gave less specific guidance. 

• PACE incoiporated gender-responsive programming into all services through a focus on safety 
and relationships, an emphasis on recognizing and building on girls’ individual strengths, and an 
awareness of the effects of trauma. 

• Girls who attend PACE tend to be low-income, and they often struggle with school and have a 
range of other health, safety, and delinquency risk factors. 

• PACE differed from the traditional school environment by offering smaller classes, access to 
frequent individual academic advising and counseling, life skills programming, and connections 
to other services in the community, such as transportation or health care. 
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Preface 


When young people drop out of school or become involved in the juvenile justice system, the 
consequences can extend far into adulthood. Thus, there is a compelling policy need to under- 
stand how to support young people who exhibit warning signs of academic failure and delin- 
quent behavior. Such behavior is often a symptom of other challenges in the lives of girls and 
boys — and girls face their own distinct challenges. Girls in the juvenile justice system are more 
likely than boys to have experienced sexual abuse and maltreatment as children, and their 
responses to trauma differ from those of boys. Yet a juvenile justice system designed for boys is 
too often ill equipped to address those issues and may only worsen girls’ problems. Gender- 
responsive programs, such as the one described in this report, were developed in recognition of 
this need. 

MDRC’s evaluation of the PACE Center for Girls offers a valuable opportunity to un- 
derstand how the gender-responsive approach translates into actual program operations. PACE 
takes a preventive approach, aiming to help troubled girls ages 1 1 to 1 8 stay in school and avoid 
involvement, or deeper involvement, with the juvenile justice system. At locations across the 
state of Florida, PACE provides academic and social services during regular school hours in a 
safe, supportive environment tailored to girls’ needs, with an emphasis on relationships, relevant 
life skills, and the cultivation of girls’ strengths. A low staff-to-participant ratio allows for 
individual attention and an awareness of each girl’s history of trauma. And, recognizing that the 
girls’ problems are often intertwined with family and peer relationships, either as cause or 
result, PACE staff members also strive to engage this larger community in the girls’ care. 

This report describes the implementation of PACE at the 14 centers that are participat- 
ing in the evaluation. The research found that PACE successfully implemented its unique model 
as planned in multiple locations. Besides detailing the program’s dissemination of its gender- 
responsive culture and services, these findings provide useful infonnation to social service 
providers who seek to replicate their own programs. In addition, the study has found that, after 
12 months, girls in PACE were more likely than girls in a control group to have received 
academic advising and mental health counseling and to have been enrolled in school. The final 
report, due in 2018, will provide experimental evidence of the impact of this gender-responsive 
program on girls’ well-being, along with a cost-effectiveness analysis. 

Gender-responsive programming has sparked federal interest and funding for research 
and development. This study will provide foundational knowledge about the effectiveness of 
this approach in helping girls stay in school and out of the court system. 

Gordon L. Berlin 

President, MDRC 
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Executive Summary 


In Florida, girls who are falling behind in school or exhibiting troubling behavior have access to 
a unique program that offers them a chance to get back on track. PACE Center for Girls 
employs what is known as a gender-responsive approach to provide both academic and exten- 
sive social services, including classes with a low student-to-teacher ratio, regular counseling 
sessions, and a life skills curriculum designed for girls. PACE, which has centers in 19 counties 
across the state, began more than 30 years ago as a program to meet the needs of girls involved 
with the juvenile justice system. The program serves middle school and high school-age girls 
who have multiple risk factors. 

These risk factors, which include individual, peer, family, school, and community char- 
acteristics, increase the likelihood that a girl will struggle in school and engage in delinquent 
behavior. Delinquency and involvement in the juvenile justice system, in turn, result in consid- 
erable personal and societal costs. Juvenile charges or detention may damage a young person’s 
relationships with friends and family, negatively affect mental health, and interrupt the academ- 
ic progress and work experience that should accumulate during adolescence. 1 And from a 
societal perspective, court and detainment costs are high. Therefore, effective prevention or 
early intervention programs that can help young people avoid involvement in the juvenile 
system and succeed in school offer a significant return on investment. 2 

Research has shown that adverse childhood experiences affect boys and girls different- 
ly. Girls have a greater incidence of depression than boys and respond differently to trauma; for 
example, girls are more likely to engage in self-medicating behaviors. 3 Their pathways into the 
justice system are also different. Girls are more often detained for nonserious offenses, such as 
truancy or violating probation, and more often enter the juvenile justice system with a history of 
physical, sexual, or emotional abuse, extreme family conflict, and neglect. 4 Gender-responsive 


Anna Aizer and Joseph J. Doyle Jr., “Juvenile Incarceration, Human Capital, and Future Crime: Evidence 
from Randomly-Assigned Judges,” Quarterly Journal of Economics 130, no. 2 (2015): 759-803. 

2 Steve Aos, Roxanne Lieb, Jim Mayfield, Mama Miller, and Annie Pennucci, Benefits and Costs of Pre- 
vention and Early Intervention Programs for Youth (Olympia: Washington State Institute for Public Policy, 
2004). 

3 Margaret A. Zahn, Robert Agnew, Diana Fishbein, Shari Miller, Donna-Marie Winn, Gayle Dakoff, 
Candace Kruttschnitt, Peggy Giordano, Denise C. Gottfredson, Allison A. Payne, Barry C. Feld, and Meda 
Chesney-Lind, Causes and Correlates of Girls’ Delinquency (Washington, DC: Office of Juvenile Justice and 
Delinquency Prevention, U.S. Department of Justice, 2010); Emily J. Salisbury and Patricia Van Voorhis, 
“Gendered Pathways: A Quantitative Investigation of Women Probationers’ Paths to Incarceration,” Criminal 
Justice and Behavior 36, no. 6 (2009): 541-566. 

4 Charlotte Lyn Bright and Melissa Jonson-Reid, “Young Adult Outcomes of Juvenile Court-Involved 
Girls,” Journal of Social Service Research 36, no. 2 (2010): 94-106; Charlotte Lyn Bright and Melissa Jonson- 

(continued) 
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approaches were developed as a response to the recognition that the current juvenile justice 
system is not well positioned to meet the particular needs of girls, as most services are based on 
the needs of boys. 5 The term “gender-responsive” thus describes treatment approaches for 
serving women and girls. Principles of gender-responsive programs include an understanding of 
the effects of trauma, a focus on relationships, and life skills and health education that is tailored 
to the lives of girls and women. 

Rigorous research on gender-responsive programming is limited, however. There has 
been national interest in understanding gender-responsive programs in the context of improving 
the juvenile system more broadly, 6 but the current literature is more robust in its description of 
concepts and principles than in its evaluation of program performance. 7 Until recently, it was 
largely unknown how gender-responsive services are implemented, how similar they are to one 
another, or how effective they are. 8 The evaluation of PACE Center for Girls — perhaps the 
largest and most well-established program of its kind — provides an opportunity to answer 
foundational questions about the implementation and effectiveness of a gender-responsive 
program. The research aims to help practitioners and policymakers better understand, and 
possibly replicate, services for at-risk girls. The evaluation is being conducted by MDRC and is 
funded mainly through the Edna McConnell Clark Foundation’s Social Innovation Fund (SIF), 
a program of the Corporation for National Community Service (CNCS), with additional funding 
provided by the Jessie Ball duPont Fund and the Healy Foundation. 

About PACE Center for Girls 

PACE Center for Girls currently operates 1 9 nonresidential, year-round program sites across the 
state of Florida. Girls eligible for PACE are between the ages of 1 1 and 1 7 and exhibit such risk 


Reid, “Onset of Juvenile Court Involvement: Exploring Gender-Specific Associations with Maltreatment and 
Poverty,” Children and Youth Services Review 30, no. 8 (2008): 914-927; Zahn et al., Causes and Correlates 
of Girls ’ Delinquency. 

5 Margaret A. Zahn, Stephanie R. Hawkins, Janet Chiancone, and Ariel Whitworth, The Girls Study Group 
— Charting the Way to Delinquency Prevention for Girls (Washington, DC: Office of Juvenile Justice and 
Delinquency Prevention, U.S. Department of Justice, 2008). 

6 Juvenile Justice and Delinquency Prevention Act, 42 U.S.C. 5633 § 242 (1992). 

7 Dana Jones Hubbard and Betsy Matthews, “Reconciling the Differences Between the ‘Gender- 
Responsive’ and ‘What Works’ Literatures to Improve Services for Girls,” Crime & Delinquency’ 54, no. 2 
(2008): 225-258. 

8 Meda Chesney-Lind, Merry Morash, and Tia Stevens, “Girls’ Troubles, Girls’ Delinquency, and Gender 
Responsive Programming: A Review,” Australian & New Zealand Journal of Criminology 4, no. 1 (2008): 
162-189; Patricia K. Kerig and Sheryl R. Schindler, “Engendering the Evidence Base: A Critical Review of the 
Conceptual and Empirical Foundations of Gender-Responsive Interventions for Girls’ Delinquency,” Laws 2, 3 
(2013): 244-282. 
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factors as exposure to abuse or violence, poor academic perfonnance, truancy, risky sexual 
behavior, substance abuse, and other stressors that may contribute to trauma and negative 
outcomes. PACE aims to get them back on track by providing services in a gender-responsive 
environment that addresses these risk factors and develops their strengths. 

Girls in this voluntary program, who live primarily at home, attend PACE daily during 
normal school hours and receive academic and extensive social services. These services include 
academic instruction and advising, a life skills curriculum, assessment and care planning, 
individual and group counseling, parental engagement, volunteer service and work readiness 
opportunities, and transition and follow-up services. Girls typically plan to attend PACE for 
approximately one year and often return to other schools in their communities to complete their 
education. 9 A low staff-to-girl ratio allows for individual attention and opportunities to build 
relationships, contributing to the girls’ sense of safety and belonging while they are in attend- 
ance. PACE centers strive to create inclusive environments in which a variety of support 
services “wrap around” each girl, and they rely on a strengths-based approach — emphasizing a 
girl’s assets rather than deficits — and an understanding of trauma and its effects when dealing 
with girls’ risky or challenging behaviors. 

The PACE Evaluation 

In response to the growing need to better understand and evaluate the services available to girls 
at risk of school failure, delinquency, substance abuse, or other poor physical and mental health 
outcomes, this study aims to provide evidence on the execution and effectiveness of the PACE 
program. The evaluation has three main components: an impact study, an implementation study, 
and a cost-effectiveness analysis. Fourteen PACE centers participated in the evaluation during 
the two-year study enrollment period, from August 2013 through October 2015. 

The impact evaluation employs a random assigmnent design. With this design, girls 
who were deemed eligible for PACE enrolled in the study and were assigned at random either 
to a program group, whose members are offered PACE services, or to a control group, whose 
members are referred to other services in the community. Between August 2013 and October 
2015, 1,134 girls were enrolled in the study (679 in the program group and 455 in the control 
group). The results of the impact study, which will be published in a future report, will provide 
infomiation on the degree to which PACE prevented negative outcomes and created positive 
opportunities for girls. 


9 In some cases, girls seek options other than returning to the public school they attended previously or 
another school in the district; for example, earning a high school equivalency diploma and gaining employ- 
ment. In rare cases, PACE centers provide a high school diploma through the local school district. 
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This report focuses on how PACE implements its gender-responsive services at each of 
the centers in the study. As noted above, few gender-responsive programs have been evaluated, 
and information on how this type of program actually operates is limited. This research also will 
inform policymakers and practitioners interested in understanding how the PACE program 
model is replicated across locations. 

The implementation study focused on answering three main questions: 

• How is PACE implemented at each center? This included a close examina- 
tion of how gender-responsive programming is provided. 

• Whom does PACE serve? This involved understanding how girls were se- 
lected to participate in the program and how girls in the study compared with 
the general population of girls in Florida. 

• How does PACE differ from other services available in the community for 
at-risk girls? 

The research team reviewed the intended implementation of services according to the 
PACE program model and compared these with the actual implementation of services. Addi- 
tional analysis examined whether implementation of the model or services varied across centers. 
These analyses draw from a rich set of qualitative and quantitative data. Quantitative data 
presented in this report include the baseline characteristics of the research sample, program 
participation data, a survey of PACE staff members, a validated classroom observation scoring 
system, and a follow-up survey to the study sample of girls (both the program and control 
groups) 12 months after study enrollment. 10 Qualitative data include staff interviews, observa- 
tions of program activities, in-person individual interviews and focus groups with current PACE 
participants, and follow-up phone interviews with program group participants. 

Key Findings 

This section summarizes findings from the implementation study. The full report provides 
additional findings, including details on how services were delivered, for each component of 
PACE’s service model. 

• The PACE program model was implemented consistently across multi- 
ple locations. Some variation in services across locations occurred be- 


10 Fielding of this survey was ongoing at the time the current report was written. Therefore, the survey re- 
sponses presented here are from girls enrolled in the study between August 2013 and March 2015, about two- 
thirds of the frill study sample. 
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cause of differences in staffing and access to resources and in program 
areas where the model provides less guidance. 

PACE takes a hybrid approach to defining its model. In addition to a written set of val- 
ues and guiding principles that describe how staff members should approach their work, PACE 
provides detailed guidance on program activities in a lengthy manual. PACE headquarters 
supports implementation through staff training, data systems, and a quality assurance process. 
The central office plays a key role, monitoring fidelity and providing guidance or support as 
needed. The research team found that individual PACE centers were implementing the model as 
intended. Staff members described in interviews how PACE’s values and guiding principles 
provided the foundation for how they did their work, and both management and direct service 
staff members reported using the manual regularly to guide program implementation. Data from 
PACE’s management information system show that girls at PACE are receiving most services 
at the prescribed frequency and intensity. 

Activities that were not specified in the manual had more variation in implementation. 
In these areas — for example, the approach counselors should use with girls — variation 
seemed to be driven largely by the experience and approach of individual staff members. 
Availability of resources was another factor. Though every center receives a basic level of 
funding through PACE’s contracts with the Florida Department of Juvenile Justice and local 
school districts , 11 each center drew on community resources to support or augment core ser- 
vices. Centers with more resources were able to offer additional services, such as health care 
and therapy on site. Centers with fewer resources had fewer staff members to dedicate to certain 
activities (for example, transition services, volunteer services, and work readiness). 

• PACE incorporated gender-responsive programming into all services 
through a distinct program culture and through specific program com- 
ponents such as assessments, life s kil l s classes, and parental engagement. 

PACE’s model incorporates many of the key tenets of gender-responsive programming 
that are cited by practitioners and researchers in the field. The implementation research found 
that PACE infused gender-responsive programming into many aspects of service delivery 
through a distinct program culture, focusing on safety and relationships, an understanding of 
trauma, and an emphasis on building girls’ individual strengths. Key aspects of gender- 
responsive programming were also incorporated into assessments, life skills, and parental 
engagement. Table ES.l provides an overview of common elements of gender-responsive 
programs and how they are put in practice at PACE. 


"PACE, Seek Excellence: 2014 Annual Report (Jacksonville, FL: PACE Center for Girls, 2014). 
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Table ES.l 


Gender-Responsive Programming Principles and 
PACE Program Components 


Category 

Principle of 

Gender-Responsive 

Programming 

PACE Program Component 

Program 

environment 

Safety 

PACE provides secure facilities, behavior management, and a 
program culture that is intended to be safe from bullying and 
trauma triggers. 


Focus on high-quality 
relationships 

Staff members focus on building positive and supportive 
relationships with the girls. Care is informed by the other key 
relationships in a girl’s life, including family relationships. 


Strengths-based 

approach 

Staff members are trained to recognize a girl’s assets and orient 
care toward building strengths rather than focusing on deficits. 


T rauma-informed 
approach 

Staff members are trained to recognize the symptoms of trauma 
and to understand how trauma can affect a girl’s behavior. Staff 
members use knowledge of a girl’s trauma history to inform care. 

Assessment 

Holistic approach to 
treatment 

PACE implements a comprehensive assessment process to 
understand a girl’s risk factors and protective factors across five 
domains: family, school, behavior, victimization, and health. 

Life skills 

Education about 
women’s health 

The Spirited Girls! life skills curriculum educates girls about 
healthy relationships and general and reproductive health. Staff 
members work with girls to address specific women’s health 
needs. 


Educational and 
vocational opportunities 

Academic services provide girls with an opportunity to catch up to 
grade level by providing individual support in small classes. 

Career exploration is provided in Spirited Girls! classes or through 
separate career classes. Staff members provide individual support 
on career planning. 


Connections to the 
community 

Volunteer service provides girls with the opportunity to connect 
with the community in a positive way. 

Parental 

engagement 

Emphasis on family 

Staff members engage a girl’s family in her care through regular 
updates on her progress and by seeking to address needs within 
the family when possible. Staff members use an awareness of 
each girl’s family dynamics to inform her care. 


SOURCES: Developed from Patricia K. Kerig and Sheryl R. Schindler, “Engendering the Evidence Base: A 
Critical Review of the Conceptual and Empirical Foundations of Gender-Responsive Interventions for Girls’ 
Delinquency,” Laws 2, 3 (2013): 244-282; and interviews with PACE staff members. 
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• PACE serves girls who tend to be low-income, to be struggling with 
school, and to have other risk factors, such as prior abuse or involve- 
ment with the juvenile justice system. 

PACE implements a thorough assessment and screening process to assess whether a 
girl meets eligibility requirements, to understand her history and risk factors, and to determine 
whether she would be a “good fit” for the program. PACE serves girls who are struggling 
academically and who exhibit a range of health, safety, and delinquency risk factors. As shown 
in Table ES.2, at the time of study enrollment, about half the study sample had been held back 
at least once, and a large portion had low school attendance. Many girls had experienced abuse 
or neglect or reported having thoughts about harming or killing themselves. A significant 
portion of the sample were sexually active. Nearly 30 percent of participants had been previous- 
ly arrested, and a majority of the sample had a family member with a criminal history. Partici- 
pants also came predominantly from low-income families and often from single-parent house- 
holds. 


• PACE differed from what girls experienced in other school settings in 
several distinct areas. Girls assigned to the program group and invited 
to attend PACE were more likely than girls in the control group to have 
been enrolled in school and to have received academic advising, counsel- 
ing, and other services in the 12-month period since study enro ll ment. 

In a review of other programs available in the communities served by PACE, none of- 
fered a similar combination of academic and social services in a gender-responsive setting. 
Some programs offered both academic and social services but without the gender-responsive 
approach. PACE’s academic services differed from those offered at public schools, which many 
control group members attended, in tenns of class size and access to regular academic advising. 
Classes were capped at 14 students, and the low student-teacher ratio allowed teachers to 
provide girls with more individual support. PACE also provided more academic advising than 
was typically provided by public schools. In the 12 months since random assignment, as shown 
in Table ES.3, girls in the program group were more likely to have been enrolled in school and 
to have received academic advising than girls in the control group. On the quality of classroom 
instruction, as measured by one common assessment tool, PACE scored similarly to public 
schools. 


Social services at PACE also distinguished it from the traditional school environment. 
Responses from the follow-up survey indicate that girls in the program group received more 
social service support than girls in the control group during the 12 months following study 
enrollment. For example, girls in the program group were 19 percentage points more likely than 
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Table ES.2 


Selected Characteristics of Sample Members at Baseline 


Characteristic (%) 

Full 

Sample 

Demographic 

Age 

11-12 

8.5 

13-14 

32.5 

15-16 

49.5 

17 or older 

9.5 

Race/ethnicity 

Black, non-Hispanic 

45.1 

Hispanic 3 

16.0 

White, non-Hispanic 

38.1 

Other 

0.8 

People participant lives with 

Two parents 

34.8 

Single parent 

51.8 

Relative 

10.6 

Other b 

2.8 

Family income 0 

$28,050 or lower 

41.2 

$28,051-$44,900 

35.5 

Above $44,900 

23.3 

Academic 

School level at time of referral to PACE 

6th grade d 

8.8 

7th-8th grade 

37.2 

9th- 10th grade 

45.3 

1 1th- 12th grade 

8.7 

Recently expelled or suspended 0 

39.6 

Has more than 15 absences in past 3 months 

41.7 

Held back at least once 

51.8 

Has a learning disability 

29.6 


(continued) 
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Table ES.2 (continued) 


Characteristic (%) 

Full 

Sample 

Delinauencv 


Ever arrested f 

27.7 

Ever been on probation 

12.6 

Has family member with criminal history® 

64.1 

Health and safety 


Ever sexually active 

44.1 

Abused/neglected 11 

38.1 

Ever had thoughts about harming/killing herself 

39.3 

Sample size 

1,134 


SOURCE: MDRC calculations based on data from the PACE management 
information system. 

NOTES: Certain characteristics listed here were captured in two different ways 
during the random assignment period, as noted below. 

a Sample members are coded as Hispanic if they answered "yes" to Hispanic 
ethnicity. 

b "Other" includes nonrelative or foster care. 

c The U.S. Department of Housing and Urban Development's guidelines were 
used in the data collection process to determine which income range the 
participant's family fell into. Since these figures could vary by county and 
household size, the ranges presented here correspond to statewide income limits 
for low income and very low income for a four-person household in Florida in 
fiscal year 2014. 

d This category includes sample members who were in fifth grade at the time of 
referral. 

e For approximately half of the sample, this was defined as being currently 
expelled or suspended. For the other half of the sample, this referred to one or 
more expulsions or suspensions in the most recent school term. 

r ln the juvenile justice system, people are not technically "arrested"; the 
terminology used is either "incurred a charge" or "referred." 

g For approximately half of the sample, this measure referred to a criminal 
record (including imprisonment, probation, parole, and house arrest) for a parent, 
guardian, or sibling of the sample member. For the other half of the sample, 
"family" included other members of the household as well. 

h For approximately half of the sample, this measure referred only to 
documented instances of abuse or neglect. For the other half of the sample, the 
measure also included suspected incidents of abuse. 
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Table ES.3 


Impacts on Service Receipt 



Program Control Difference 


Outcome (%) 

Group 

Group 

(Impact) 

P-Value 

Academic Service Receipt 

Ever enrolled in a school or educational program 

99.1 

92.9 

6 2 *** 

0.000 

Received academic advising 

81.1 

67.6 

13 5 *** 

0.000 

Frequency of academic advising sessions 

More than once per month 

38.0 

33.8 

4.2 

0.293 

Once per month 

19.6 

11.5 

g 2 *** 

0.009 

1-3 times per year 

23.4 

22.4 

1.0 

0.770 

Never 

19.0 

32.4 

-13 4 *** 

0.000 

Social Service Receipt 

Received help finding services in the community 

38.5 

23.3 

15 2 *** 

0.000 

Received mental health counseling or therapy 

64.2 

45.5 

lg 2 *** 

0.000 

Frequency of counseling or therapy sessions 

Once per week or more 

39.3 

23.9 

15 4 *** 

0.000 

1-3 times per month 

19.9 

13.4 

6.5 ** 

0.043 

Less than once per month 

4.6 

8.0 

-3.5 * 

0.081 

Never 

36.2 

54.6 

-18.5 *** 

0.000 

Received help related to sexuality, sex, 

or sexual and reproductive health 

72.0 

57.8 

14 3 *** 

0.000 

Received help related to social and emotional skills 

80.3 

63.0 

12 3 *** 

0.000 

Sample size (total = 668 ) 

407 

261 




SOURCE: MDRC calculations based on girls' responses to the PACE evaluation 12-month follow-up 
survey. 

NOTES: The sample size reported here is based on responses to the follow-up survey among girls 
randomly assigned between August 2013 and March 2015. Due to missing values, the number of girls 
included varies by outcome. 

Results in this table are regression-adjusted, controlling for pre -random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
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girls in the control group to have received mental health counseling or therapy. They were also 
more likely to have received help connecting to other services in the community, such as 
transportation or housing, than the control group. And PACE offers more life skills program- 
ming focused on the needs and perspectives of girls than is typically available in public schools. 
For example, program group girls were more likely to report that they had received help related 
to sex and reproductive health than the control group (72 percent compared with 58 percent). 
PACE also offers follow-up services to girls after they leave the program, though the intensity 
of services offered varied among the centers. 

Summary 

For girls at risk of an array of negative outcomes, this report highlights the ways that PACE is 
different from the other options available to girls in the communities where PACE operates. 
Few programs offer the same combination of services. The impact on reported service receipt 
bears this out. 

PACE is also an example of a program that provides similar services and a consistent 
culture in multiple locations in diverse communities. PACE’s approach — specifying its model 
through both principles and a manual, supporting staff members through training, and using 
data to monitor implementation and fidelity — offers lessons for the field more broadly. 
Variation tended to occur in areas where the program model was not specified — for example, 
the approaches counselors took with girls. Finding the balance between specification and 
flexibility is an ongoing tension in the replication of human service programs. 

The implementation study of PACE also offers an opportunity to understand how a 
gender-responsive program actually operates, an area where current research is lacking. The 
report describes how PACE creates a gender-responsive culture as a framework for providing 
its services. The culture serves as the foundation for its gender-responsive programming and is 
infused into all aspects of program delivery. Building from this culture, PACE offers a combi- 
nation of services that is hypothesized to meet the specific needs of at-risk girls. 

This implementation report is one in a series of publications from the PACE evaluation 
that will add to the evidence base regarding gender-responsive programming and its effective- 
ness. In early 2016, a research brief provided an introduction to the study. Another brief, 
released as a companion to the current report, delves further into the history and literature 
around gender-responsive programming, using PACE implementation as a case study. And in 
2018, a final report will present the results of the impact study and a cost-effectiveness analysis, 
which will evaluate the costs of PACE in the context of its outcomes for girls. 
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Chapter 1 

Introduction 


In Florida, middle and high school-age girls who are falling behind academically or exhibiting 
behavioral problems may have an opportunity to get back on track through a unique program: 
PACE Center for Girls. PACE centers, located in 19 counties across the state, employ a “gen- 
der-responsive” approach to provide both academic and extensive social services. Girls attend 
daily and year-round. PACE offers classes with a low student-to-teacher ratio, regular counsel- 
ing sessions, and a life skills curriculum designed for girls, among other services. PACE began 
more than 30 years ago as a program to meet the needs of girls involved with the juvenile 
justice system. The program focuses on serving girls who exhibit multiple risk factors for 
delinquency. 

These risk factors, which include individual, peer, family, school, and community char- 
acteristics, increase the likelihood that a girl will struggle in school and engage in delinquent 
behavior. Delinquency and involvement in the juvenile justice system, in turn, result in consid- 
erable personal and societal costs. Juvenile justice involvement may damage a young person’s 
relationships with friends and family, negatively affect mental health, and interrupt the academ- 
ic progress and work experience that should accumulate during adolescence. 1 And from a 
societal perspective, court and detainment costs are high. Therefore, effective prevention or 
early intervention programs that help young people avoid involvement in the juvenile system 
can offer a significant return on investment. 2 Policymakers seek to identify and evaluate 
promising approaches. 

Girls at risk of entering the juvenile court system, or those who are involved in the sys- 
tem already, have profiles that differ from those of their male counterparts: Girls are more often 
taken in for nonserious offenses, such as truancy or violating probation, and they are more likely 
to enter the system with a history of physical, sexual, or emotional abuse, extreme family 
conflict, or neglect. 3 Girls, more broadly, also have a greater incidence of depression than boys. 4 
Girls and boys tend to respond differently to trauma, and there is a stronger association between 
traumatic stress and mental health problems among girls. 5 As most services are based on the 
needs of boys, the current juvenile justice system is not well positioned to meet the particular 


Aixer and Doyle (2015). 

2 Aos et al. (2004). 

’Bright and Jonson-Reid (2008, 2010). 
4 Kleinfeld (2009). 

5 Zahn et al. (2010). 
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needs of girls. According to a report by the Georgetown Law Center on Poverty, Inequality and 
Public Policy, “the juvenile justice system only exacerbates [the girls’] problems by failing to 
provide girls with services at the time when they need them most.” 6 

Gender-responsive prevention programs offer a promising way to address girls’ unique 
needs. 7 Specifically, “gender-responsive” describes treatment approaches for serving only girls 
and women, based on the understanding that the default approach is designed for boys and 
men. 8 While good gender-responsive services share the basics of any good services — such as a 
well-trained staff and solid treatment approaches 9 — they are distinctive in bringing an aware- 
ness of girls’ particular development and gender-specific issues into the program. (Chapter 4 
provides more detailed information on gender-responsive programming.) Federal and local 
policymakers have lent their support for this type of program. A 1992 amendment to the 
Juvenile Justice and Delinquency Prevention Act provided funding for research and develop- 
ment of gender-responsive services. 10 In 2004, the U.S. Department of Justice’s Office of 
Juvenile Justice and Delinquency Prevention (OJJDP) created a Girls Study Group to further 
the research base around programming for girls. 11 More recently, OJJDP partnered with a 
national organization to create the National Girls Initiative, which provides training, technical 
assistance, and other resources to programs serving this population. 12 And in 2015, OJJDP 
released a statement affirming its commitment to provide funding for research about girls in the 
juvenile justice system. 13 Thus, at the federal level, gender-responsive services are considered 
an important part of the service array. 

Even with this growing interest, rigorous research on gender-responsive programs re- 
mains scant. The current literature is robust in its description of concepts and principles, but 
until recently, it was largely unknown how gender-responsive services are implemented, how 
similar programs are to one another, or how effective they are. 14 Researchers have characterized 
the empirical literature in this area as “limited and inadequate” 15 and “in its infancy.” 16 


6 Watson and Edelman (2013), p. ii. 

7 Office of Juvenile Justice and Delinquency Prevention (2016a). 

8 Covington and Bloom (2006); Zahn, Hawkins, Chiancone, and Whitworth (2008). 

9 Maniglia (1998). 

10 Office of Juvenile Justice and Delinquency Prevention (2015, 2016a). 

"Office of Juvenile Justice and Delinquency Prevention (2016b). 

12 National Crittenton Foundation (2016). The National Girls Initiative was formerly known as the Nation- 
al Girls Institute. 

"Office of Juvenile Justice and Delinquency Prevention (2016a, 2016b). 

14 Chesney-Lind, Morash, and Stevens (2008); Kerig and Schindler (2013); Hubbard and Matthews (2008). 
15 Chesney-Lind, Morash, and Stevens (2008), p. 183. 

"’Bright and Jonson-Reid (2010), p. 103. 
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The evaluation of PACE Center for Girls — perhaps the largest and most well- 
established gender-responsive program of its kind — provides an opportunity to assess this 
approach. The evaluation is answering foundational questions about the implementation and 
effectiveness of a gender-responsive program, helping practitioners and policymakers better 
understand these services for at-risk girls. The evaluation is being conducted by MDRC and is 
funded mainly through the Edna McConnell Clark Foundation’s Social Innovation Fund (SIF), 
a program of the Corporation for National Community Service (CNCS), with additional funding 
provided by the Jessie Ball duPont Fund and the Healy Foundation. (See Box 1.1 for further 
infomiation about the SIF.) 

This report specifically provides infomiation on the PACE program model and how 
PACE implements that model. The key findings from this report include the following: 

• The PACE program model was consistently implemented across multiple lo- 
cations, with a core set of similar services. Services differed somewhat across 
locations in program areas where the PACE model does not provide detailed 
guidance or when resources were limited. 

• PACE incorporated gender-responsive programming into its services by cre- 
ating a safe, relationship-focused environment and by using tools and ap- 
proaches that fit within the gender-responsive principles. 

• PACE’s environment and combination of services differed from what girls 
experienced in a more traditional school setting. Girls invited to attend PACE 
were more likely than a control group to have enrolled in school, received 
academic advising, participated in counseling, and received other services in 
the 12-month period since study enrollment. 


PACE Center for Girls 

PACE Center for Girls currently operates 19 nonresidential program locations across the state 
of Florida. Applicants to this voluntary program are 1 1 to 17 years old and are typically strug- 
gling academically and may have behavioral problems. PACE aims to help by providing 
services in a gender-responsive environment that develops their strengths and addresses their 
risk factors, such as exposure to abuse or violence, poor academic performance, truancy, risky 
sexual behavior, substance abuse, and other family or social stressors. 

PACE locations are referred to as “centers” — to differentiate them from the school setting that 
girls were usually coming from — and offer services year-round. Girls travel daily to the 
program during nonnal school hours and receive academic services and extensive social 
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Box 1.1 


The Edna McConnell Clark Foundation (EMCF) 

Social Innovation Fund 

The Social Innovation Fund (S1F) — an initiative enacted under the Edward M. Kennedy 
Serve America Act — directs millions of dollars in public-private funds to expand effective 
solutions in three issue areas: economic opportunity, healthy futures, and youth development 
and school support. This work seeks to create a catalog of proven approaches that can be 
replicated in communities across the country. The SIF generates a 3:1 private-public match, 
sets a high standard for evidence, empowers communities to identify solutions to social 
problems, and creates an incentive for grant-making organizations to target funding more 
effectively to promising programs. Administered by the federal Corporation for National and 
Community Service (CNCS), the SIF is part of the government’s broader agenda to redefine 
how evidence, innovation, service, and public-private cooperation can be used to tackle urgent 
social challenges. 

The Edna McConnell Clark Foundation, in collaboration with MDRC and The Bridgespan 
Group, is leading a SIF project that aims to expand the pool of organizations with proven 
programs that can help low-income young people make the transition to productive adulthood. 
The project focuses particularly on young people who are at greatest risk of failing or dropping 
out of school or of not finding work; who are involved or likely to become involved in the 
foster care or juvenile justice system; or who are engaging in risky behavior, such as criminal 
activity or teenage pregnancy. 

EMCF, with its partners MDRC and Bridgespan, selected an initial group of nine programs 
and a second group of three programs to receive SIF grants: BELL (Building Educated Lead- 
ers for Life), the Center for Employment Opportunities, Children’s Aid Society-Carrera 
Adolescent Pregnancy Prevention Program, Children’s Home Society of North Carolina, 
Communities In Schools, Gateway to College Network, PACE Center for Girls, Reading 
Partners, The SEED Foundation, WINGS for Kids, Youth Guidance, and Children’s Institute, 
Inc. These organizations were selected through a competitive selection process based on prior 
evidence of impacts on economically disadvantaged young people, a track record of serving 
young people in communities of need, strong leadership and a potential for growth, and the 
financial and operational capabilities necessary to expand to a large scale. 

The EMCF Social Innovation Fund initiative is called the “True North Fund” and includes 
support from CNCS and 15 private coinvestors: The Edna McConnell Clark Foundation, The 
An nie E. Casey Foundation, The Duke Endowment, The William and Flora Hewlett Founda- 
tion, The JPB Foundation, George Kaiser Family Foundation, The Kresge Foundation, Open 
Society Foundations, Penzance Foundation, The Samberg Family Foundation, The Charles 
and Lynn Schusteiman Family Foundation, The Starr Foundation, Tipping Point Community, 
The Wallace Foundation, and Weingart Foundation. 




services. Girls typically plan to attend PACE for about one year and often return to schools in 
their communities to complete their education. 17 A low staff-to-girl ratio allows for individual 
attention and opportunities to build relationships, contributing to the girls’ sense of safety and 
belonging while they are in attendance. PACE centers strive to create inclusive environments in 
which a variety of support services “wrap around” each girl, with attention to each girl’s 
individual strengths. Figure 1.1 shows program inputs, the services provided, and the intended 
outcomes. 

The gender-responsive PACE program model includes: 

• Academic instruction. Girls receive daily middle school- or high school- 
level instruction in a small class setting. Individual academic plans guide 
progress, which is monitored through regular advising sessions. 

• Life skills curriculum. Girls regularly attend a Spirited Girls! class that cov- 
ers six domains believed to be essential for girls’ healthy development: phys- 
ical, emotional, intellectual, relational, sexual, and spiritual. 

• Individual assessment and care planning. Assessments of each girl’s needs 
are used to create tailored plans for the girl’s time at PACE. Staff members 
meet regularly to share infomiation and review progress. Staff members refer 
girls to services outside PACE as needed. 

• Individual and group counseling. Girls attend frequent individual sessions 
with counselors and regular psychoeducational group sessions. 

• Parental engagement. Program staff members maintain parental engage- 
ment through initial home visits, monthly progress reports, office sessions, 
and phone contact. 

• Volunteer service and work readiness. The centers provide volunteer ser- 
vice opportunities, career exploration, and work readiness training. 

• Transition and follow-up services. Additional support is available for girls 
as they make the transition out of PACE and back to their home school or 
another appropriate placement. Staff members also check in with girls at reg- 
ular intervals after they leave the program to provide services or referrals, if 
needed. 


l7 In some cases, girls seek options other than returning to their previous public school or another school in 
the district; for example, earning a high school equivalency diploma and gaining employment. In rare cases, 
PACE centers can provide a high school diploma through the local school district. 
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Figure 1.1 

PACE Center for Girls Logic Model 


Outcomes 
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characteristics (e.g., H (e.g., leadership, staff turnover, culture, H (e.g., neighborhood conditions, 
level of risk, readiness) H and climate) H alternative services) 








Each center is led by an executive director, with additional leadership provided by a 
program director and other managers. 18 A social service manager oversees a counseling staff, 
and an academic manager oversees the academic and life skills teachers. The statewide PACE 
headquarters provides supportive services to all the centers, assisting with fundraising, finance, 
human resources, legal matters, training, technical assistance, and infomiation technology. The 
headquarters management team also advocates for resources and public policy at the state level 
and coordinates regular meetings of the center staff to allow for infonnation sharing statewide. 

PACE receives more than two-thirds of its funding through two sources: the state’s ed- 
ucational system and the Florida Department of Juvenile Justice (DJJ). 19 For academic services, 
the centers contract directly with the local school districts to receive per pupil funding that 
supports a full academic school day. 20 The DJJ funding supports the social services provided to 
each girl and her family, as well as facility expenses; this funding comes with specific eligibility 
criteria and limits a girl’s length of stay at PACE to 15 months. 21 Additional contributions come 
from federal and state grants as well as local grants from public, corporate, and private sources. 

The PACE Evaluation 

In response to the need to understand and evaluate the services available to girls at risk of an 
array of negative outcomes, this study aims to provide evidence on implementation and effec- 
tiveness of the well-established PACE program. The evaluation has three main components: an 
implementation study, an impact study, and a cost-effectiveness analysis. In early 2016, a 
research brief provided an introduction to the study. 22 A second brief, focused on gender- 
responsive programming, serves as a companion to this implementation report. And in 2018, a 
final report will present the results of the impact study and evaluate the costs of PACE in the 
context of its outcomes for girls. 

Fourteen PACE centers actively participated in the evaluation. 23 Typically, PACE cen- 
ters serve a county-wide area. Figure 1.2 shows the location of the PACE centers by population 


18 Not all PACE centers are staffed with a program director. In some cases, a program director’s responsi- 
bilities are divided between the executive director and other managers. 

19 PACE (2014). 

20 Specifics of these contracts vary by center. Contracts may provide for academic materials, part-time 
staff, or transportation, among other items. 

“'This 15-month maximum length of stay went into effect during the study period. PACE can appeal to 
DJJ in individual cases to ask for an extension beyond this 15-month period. 

22 Millenky and Mage (2016). 

23 As shown in Figure 1.2, at the time of this report’s publication, 19 PACE centers operated in Florida. 
Two centers opened after the evaluation began and thus were not eligible to be included; two other centers that 

(continued) 
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Figure 1.2 

Population Map of Florida with PACE Centers 



People per square mile 


o PACE centers in evaluation 
■' ) PACE centers not in evaluation 

Outlines indicate counties 
with PACE centers. 


SOURCES: Population density data are from the U.S. Census Bureau. PACE locations are from PACE 
(2016). 


density. Lower-density counties tended to have higher poverty rates. 24 Table 1.1 presents 
selected characteristics of the participating centers. Most centers had the capacity to serve about 


were initially slated to participate in the study were excluded because of low enrollment during the study 
period; and one rural center was not included because of ethical concerns about a lack of other services in the 
community that would be available to a control group. 

24 Florida Legislative Office of Economic and Demographic Research (2011). 


Table 1.1 


Characteristics of PACE Centers in the Study 


Characteristic 

Mean 

Min 

Max 

Executive director tenure a,b (years) 

7.5 

0.2 

22.6 

Staff tenure 6 (years) 

4.0 

1.9 

8.1 

Y ears in community 

17.9 

7.0 

28.0 

Number of program slots 

54.3 

41.0 

81.0 

Number of staff members 

18.4 

13.0 

33.0 

Staff-to-slot ratio 

1:3 

1:4 

1:2.33 

Cost per slot c ($) 

34,486 

27,975 

39,596 

Sample size 



14 


SOURCES: MDRC calculations based on responses to the PACE evaluation staff survey, 
MDRC site visit data, and program information provided by PACE. 

NOTES: a This measure is based on data from the PACE evaluation staff survey (N =13). 
b T enure refers to years at current center. 

c This is an average of program costs for fiscal years 2014 and 2015. 


50 girls at a time, though the number of program slots ranged from about 40 to 80. 25 Staffing 
varied accordingly, based on the center size. Nearly all the participating centers had operated for 
at least 15 years. 

For the impact analysis, the evaluation employed a random assignment design. Girls 
who were deemed eligible for PACE enrolled in the study and were assigned at random either 
to a program group, whose members were offered PACE services, or to a control group, whose 
members received referrals to other services in the community. The program’s existing applica- 
tion and screening processes were used to determine eligibility, and 1,134 girls were enrolled in 
the study between August 2013 and October 2015 (679 in the program group and 455 in the 
control group). 26 (See Appendix C for further infonnation about the random assignment 

~ 5 This represents the average number of program slots during the two-year study enrollment period, from 
summer 2013 through summer 2015. 

2 6 All girls who applied and were eligible for PACE during the study period were included in the study, 
except girls who had previously attended PACE, siblings of current or recent PACE participants, and girls in 

(continued) 
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procedures.) The impact analysis will use a 12-month follow-up survey and administrative 
records to examine outcomes for girls in each research group. Follow-up data collection 
activities are the same for the program and control groups. Key outcomes are academic pro- 
gress, academic engagement, juvenile justice involvement, healthy relationships, and risky 
behavior (such as high-risk sexual activity and substance abuse). Findings from the impact 
study will indicate whether PACE prevented negative outcomes and created positive opportuni- 
ties for girls in and out of school. 27 

Focusing mainly on implementation, the current report provides detailed information 
about the services that PACE provides, how those services are delivered, and whether services 
vary across PACE centers. As noted earlier, research on implementation and effectiveness of 
gender-responsive programs is quite limited. The current study addresses this gap and may also 
inform policymakers and practitioners who are interested in understanding how a program 
model is implemented across a number of locations. The implementation analysis draws on rich 
data, such as multiday visits to each participating center, surveys, and program participation 
records. 

• Site visits: Members of the research team visited each of the participating 
centers to interview program staff members at all levels, conduct interviews 
and focus groups with girls, speak to local board members and stakeholders, 
and observe classrooms and staff meetings. 28 These visits occurred during the 
study enrollment period, between May 2014 and March 2015. 

• Staff survey: A web-based survey was administered to all permanent pro- 
gram staff in the participating programs between October 2014 and March 
2015. This survey gathered infonnation about staff roles and backgrounds 
and about PACE’s organizational culture. 29 

• 12-month follow-up survey of girls: A lengthy follow-up survey was ad- 
ministered by phone or in person to study sample members approximately 


state custody through the Florida Department of Children and Families. In addition, MDRC granted each 
center a small number of “hardship passes” to serve girls with specific circumstances who PACE believed 
could not be served well elsewhere in the community. These girls were not a part of the study. 

27 As a condition of funding, the SIF requires a “large, well-designed and well-implemented randomized 
controlled, multisite trial” to provide a “strong level of evidence” (Corporation for National and Community 
Service 2014, p. 3). The impact study follows these guidelines. 

28 Classroom observations used the CLASS-Secondary tool, which is described in more detail in Chapter 5. 
“ ’interns and temporary staff members did not complete the survey. The response rate for the survey was 
high (91 percent). Information on organizational culture was collected using the 105-item Organizational 
Social Context instrument (Glisson, Green, and Williams 2012). Management-level staff members were not 
asked this set of questions. 
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one year after they applied to PACE and enrolled in the study. 30 The infor- 
mation collected in this follow-up survey will be used mainly in the impact 
analysis, but responses from the program group girls about their experiences 
with the PACE program are included in the current report. On average, girls 
in the program group responded to this survey approximately four months af- 
ter leaving PACE. In addition, the survey provides infonnation, presented in 
Chapters 5 and 6, on services received over the one-year period by both the 
program and control groups. 

• Program participation data: PACE collects extensive data in a centrally 
managed management infonnation system (MIS) at its state headquarters. 31 
The data are the most complete source of infonnation on the level and types 
of services girls receive while at PACE. They include baseline demographic 
and risk factor infonnation (collected during the program application pro- 
cess) and extensive program participation details. As discussed in this report, 
program staff members regularly enter infonnation about girls’ attendance 
and receipt of services throughout the girls’ time at PACE. See Appendix C 
for further infonnation. 

• Interviews with PACE management and stakeholders: Members of the 
research team interviewed the founder of PACE and key members of the 
PACE leadership team. These interviews occuned in January 2015 and fo- 
cused on understanding the history of PACE, the responsibilities of PACE’s 
central state office, PACE’s funding structure, and PACE’s advocacy role 
within Florida and at the national level. 

• Follow-up interviews with girls and parents: Girls who completed the 12- 
month follow-up survey and their parents or guardians were asked to partici- 
pate in an additional interview. Among those who agreed, girls and parents 
were contacted between July 2015 and March 2016. A total of 52 girls and 


30 Fielding of this survey was ongoing at the time the current report was written. Therefore, the survey re- 
sponses presented throughout the report are from girls enrolled in the study between August 2013 and March 
2015, about two-thirds of the full study sample. While all sample members were approached to complete the 
survey, not all girls could be reached or agreed to participate. Specifically, the response rate was 71 percent 
overall; 407 girls in the program group (73 percent) and 261 in the control group (69 percent) responded to the 
survey. 

31 PACE uses Social Solutions ETO (Efforts to Outcomes) software for its MIS. 
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40 parents were interviewed by telephone in an in-depth, semistructured in- 
terview about their experiences with PACE and other schooling or services. 32 


Overview of Chapters 

The remainder of this report presents the findings from the implementation study. Chapter 2 
provides background information and findings on the overall implementation of the PACE 
model. Chapter 3 presents information on girls participating in the study and the intake applica- 
tion process at PACE. Chapter 4 provides information on gender-responsive programming and 
the PACE environment. Chapters 5 and 6 discuss PACE’s academic and social services, 
respectively, including a comparison of the services received by the program and control 
groups. Chapter 7 summarizes the key findings and discusses the ongoing work at PACE and 
for the evaluation. 


32 Girls were selected to represent the different center locations and both program and control groups, as 
much as possible. The pool of parents was smaller, so all parents who agreed to participate were contacted. 
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Chapter 2 

PACE Implementation 


This chapter describes how PACE implements its program model and how it ensures that 
centers provide services as designed. The chapter first describes how PACE codifies its model 
and how staff members are trained and supported in service delivery, then presents evaluation 
findings on fidelity to the model. Subsequent chapters build on this context and go into greater 
detail about PACE implementation and the variation of services across centers. This chapter 
presents the following key findings on implementation: 

• PACE defines its model through a combination of general principles and a 
highly specific manual. Through ongoing staff training and quality assurance 
initiatives, PACE headquarters supports individual centers in their implemen- 
tation of the model to ensure consistency. As a result of these efforts, PACE 
centers are similar in the types and quantity of services they provide. 

• PACE centers had high fidelity to the aspects of the model specified by 
PACE headquarters. The few instances where fidelity was lower occurred at 
centers that had lower staff capacity or had struggled with recent staff turn- 
over. 

• Variation in implementation mostly occurred in service areas where the pro- 
gram model was not specific and the staff was given flexibility in how to im- 
plement the program. Differences in services provided by centers were also 
associated with access to local resources. 


Implementation of the PACE Model 

In order to assess implementation of a program, researchers must understand both the program 
model and how its developers disseminate that model to the staff. Program developers can take 
a variety of approaches to dissemination. One approach is described as “principle-based,” in 
which programs define a set of principles, or broad guidelines, for the activities that the staff 
should carry out. These principles are based on the program’s theory of change. Although the 
activities are defined, principle-based models provide staff members with flexibility and 
discretion in how they implement individual components. At the other end of the spectrum are 
“manualized” program models. In these models, each program activity is defined in detail, so 
staff members generally have less flexibility in how program components are implemented. 
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PACE takes a hybrid approach to defining its model, combining principle-based and 
manualized approaches. On the principle-based side, PACE has a written set of values and 
guiding principles that describe how staff members should approach their work. For example, 
the “Focus on Strengths” principle states: “We look to identify strengths in our girls, their 
families, our staff and supporters. Using these strengths as our foundation, we build strong, 
confident, productive community partnerships.” During interviews, staff members described the 
values and guiding principles as providing the framework for how services are delivered, how 
decisions are made, and how staff members interact with each other and with families. How 
PACE puts its principles into operation is discussed in greater depth in Chapter 4. 

Specific guidance on program activities is provided in the PACE policy and procedures 
manual, a document of nearly 400 pages that covers policies from staff training to required 
staff-to-girl ratios to field trip logistics. Box 2.1 provides an example of the content of the 
manual. Staff members described the manual as central to staff training and decision-making. 
New hires read the manual upon starting work at PACE. Management and direct service staff 
used the manual regularly to answer questions that came up in day-to-day work, such as how to 
respond to chronic absenteeism. Managers also said that the policy manual gave them credibil- 
ity when giving guidance to other staff members because they could point to a decision as 
following policy rather than being subjective. Important policies and changes passed down from 
headquarters were discussed at staff meetings. 

Although the manual offers specifics about what activities should occur within a PACE 
center and how often, the manual is less focused on the content of services or how services 
should be provided. For example, though the manual specifies that girls should meet every other 
week with their counselors to review their care plans, there is no policy in the manual outlining 
the approach that counselors should take in their meetings with girls. One senior manager at 
PACE described the limitations of the PACE manual in this way: “You can follow the letter of 
the policy and be accomplishing nothing. Or you can follow the letter of the policy and be 
making great progress. There’s lots of variability there.” 

PACE’s staff recruitment and training practices also support implementation of the pro- 
gram at the centers. Job descriptions are created at headquarters, so the required qualifications 
and job description for a counselor or a teacher are the same across all centers. Staff training is 
mostly delivered by each center’s own staff, with PACE headquarters providing guidance and 
developing some specific training modules. Staff members who work directly with the girls on 
a daily basis (such as teachers and counselors) must complete 80 hours of training when they 
first join PACE, plus 40 hours of annual training in subsequent years. New staff members 
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Box 2.1 


Excerpt from PACE Policy and Procedures Manual 

Policy #3.11 Policy Title: Staff to Girl Ratio 

Revised Date: 07/01/2013 
Original Date: 09/01/1992 

Policy 

PACE will generally adhere to a 1 : 12 staff to girl ratio. 

Purpose 

• To provide care management services and academic instruction. 

• To provide individualized attention to each girl. 

Procedure 

A. Upon enrollment to the program, a girl will be assigned a Teacher Advisor, advisee group 
and a social services support staff member. 

B. The [staff to girl] ratio of academic classes may be 1:14 in order to meet the following 
needs: 

1. To enable the program to effectively meet girls’ academic needs by offering classes to 
girls based on need and interest. 

2. To meet academic special needs by placing a small grouping of girls in a class, thereby 
increasing the number in other classes. 

3. To enable classes to be staffed in the case of teacher absence and the unavailability of a 
substitute. 

C. Social service groups will generally adhere to the 1:12 ratio. Some groups may exceed this 
ratio to meet the needs of the girls. 

D. The number of girls assigned to a specific Social Service Support Staff will be based on 
the individual Center’s organizational chart. 

E. The 1:12 ratio will be maintained during all program activities not mentioned above. The 
ratio should be reduced when participating in activities outside of the Center. 

F. A group list can be viewed and/or printed from the ETO system to ensure appropriate staff 
to girl ratio. 

G. Any exception to the policy must have prior approval from the Chief Program Officer or 
designee. 

SOURCE: PACE policy and procedures manual. 


15 




receive training on the PACE model, gender-responsive programming, and trauma-informed 
care. 1 Each center develops an annual “master training plan” to ensure that training require- 
ments are met; many of the training requirements are passed down through the Florida Depart- 
ment of Juvenile Justice (DJJ) or school district contracts. Although the content of the required 
training is similar, centers can tailor training sessions to suit their own needs; for example, one 
center provided staff training on human trafficking when the center saw an increase in the 
number of girls affected by it. 

PACE’s data systems play a key role in supporting implementation. The policy and 
procedures manual stipulates the frequency and intensity of all key services. These services are 
tracked for each girl in PACE’s Efforts-to-Outcomes (ETO) database. Counselors and teachers 
have the primary responsibility for entering data into the database. Managers regularly audit 
these entries to ensure that staff members are including the proper information in their entries 
and that girls are receiving services at the prescribed frequency. The focus on data was apparent 
in staff interviews and the staff survey. Counselors reported that they spent almost a quarter of 
their time on paperwork and data entry responsibilities; on average, teachers reported spending 
13 percent of their time on those tasks. 

PACE headquarters further supports implementation through organization-wide train- 
ing and model development activities, including peer learning, and takes the lead role in quality 
improvement efforts around identified needs. Examples include creating working groups that 
draw from both headquarters and center staff to update the Spirited Girls! curriculum or refine 
eligibility guidelines. 

Finally, PACE headquarters plays an important role in monitoring implementation. At 
the time of MDRC’s implementation visits, PACE was using an extensive pen-and-paper 
monitoring instrument developed from the policy and procedures manual that focused on 
ensuring that centers were in compliance with PACE and DJJ policies. 2 The instrument was 
administered by headquarters staff members and DJJ staff members during a two-day annual 
visit to each center. Activities during the visits included the review of documents, review of a 
random selection of entries into PACE’s management information system, and interviews with 
girls. If the monitoring visit revealed areas where the center was not in compliance with the 
model, headquarters and the center would together create a plan of action to address the issue. 


'Chapter 4 provides details on PACE’s gender-responsive programming and approach to trauma-informed 
care. In trauma-informed care, staff members are trained to recognize the symptoms of trauma and to under- 
stand its effects on behavior. 

2 Headquarters staff members were in the process of changing this monitoring method at the time of 
MDRC’s interviews. 
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Implementation Fidelity and Variation Across PACE Centers 

The implementation research sought to understand how closely individual centers implemented 
the PACE model and the degree to which there were differences in implementation across the 
centers. These lines of inquiry were important for two main reasons. First, because the research 
sample is pooled across centers for the impact analysis, it is important to understand whether 
girls in the study received similar services regardless of which center they attended, in order to 
interpret the results. Second, policymakers, practitioners, and researchers are interested in 
successfully replicating program models in new contexts. PACE’s approach to implementation 
may offer lessons for ensuring that multiple providers of a program adhere to an intended 
intervention. 

The MDRC evaluation team developed its own tool to assess the fidelity of PACE cen- 
ters to the core program model. The tool was based on PACE's manual, with an additional 
section on gender-responsive programming. MDRC researchers used data gathered during site 
visits and reviewed documents provided by the centers to rate each center on fidelity to 48 
items. The items were divided into six domains: 

• Staffing, including training and communication among staff (5 items) 

• Program culture, including program environment, incentive systems, and fa- 
cilities (8 items) 

• Social services, including intake, assessment, and counseling (15 items) 

• Transition services, including follow-up services (5 items) 

• Academics, including classes and academic advising (9 items) 

• Gender-responsive programming (6 items) 

For each item at each center, site visitors recorded a rating of “fidelity,” “partial fideli- 
ty,” “no fidelity,” or “unable to observe.” Site visitors were given guidance on what benchmarks 
needed to be met for each item to achieve a rating of “fidelity” or “partial fidelity.” Each 
completed tool was reviewed by the lead implementation researcher to ensure consistency 
across ratings. 

Figure 2.1 shows that fidelity in PACE centers was strong overall. Across all centers 
and domains (a total of 672 ratings), the rating “fidelity” was given 77 percent of the time. 
Across all centers and domains, about 3 percent of items were rated “no fidelity.” The figure 
shows that when broken out by domain, “partial fidelity” or “no fidelity” ratings were more 
likely to occur in academic services or transition services. These two areas, discussed in 
Chapters 5 and 6 of this report, were areas where staffing was less stable. 
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Figure 2.1 

Distribution of Program Fidelity Ratings by Domain 



Rating 

■ No fidelity 

□ Partial fidelity 

□ Fidelity 


SOURCE: Calculations based on MDRC site visit data. 

NOTES: Fourteen centers were rated on 48 items organized across six domains. For each item, centers earned a 
rating of either "fidelity," "partial fidelity," or "no fidelity." The percentages shown for each domain are across 
all centers and items. For example, for academic services, of the 126 ratings given in that domain (9 items x 14 
centers), 70 percent were "fidelity." 


The fidelity analysis provides one perspective on how similarly PACE centers are im- 
plementing the model, but because the evaluation tool was primarily based on the PACE 
manual, it is limited in terms of assessing the content or quality of services received by the girls 
in different centers. Counseling approaches, as noted earlier, are not specified in the manual. 
Interviews with the staff at each center provided further information about service content and 
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quality that were not reflected in the fidelity instrument. Overall, the research found there was 
variation in how individual centers provided services, and variation within centers in how 
individual staff members delivered services. 

In the areas where staff members had discretion in how they implemented the model, 
variation in services was related to the experience and approach of those individuals. As 
discussed in Chapter 5, analysis of data from a validated classroom observation tool found that 
teacher quality varied across teachers, rather than across centers. Similarly, as described in 
Chapter 6, interviews with counselors indicated that the approach a counselor took with a girl 
depended largely on the girl’s individual needs and the counselor’s background and training. 

At the center level, differences in implementation were primarily associated with the 
availability of resources. Each center drew on local resources to support its programs and would 
use these resources to support core services or augment the services it was able to provide. The 
fidelity analysis found that lower fidelity to the program model was most common in the areas 
of transition services and, within academic services, volunteer service and work readiness. 
Centers with lower fidelity in these areas tended to have fewer staff resources to dedicate to 
these activities, or had recently experienced staff turnover in these roles. A few centers were 
able to draw on local resources to provide services beyond PACE’s core model, such as on-site 
access to health care and mental health therapy. 

This chapter provides a framework in which to understand the findings presented in 
Chapters 4 to 6, which examine how PACE centers implemented each core service component. 
Each chapter discusses how services were intended to be provided according to the PACE 
model and PACE’s implementation plan and what services actually were provided, drawing on 
observations, interviews, and data from PACE’s management infomiation system. Areas of 
variation in implementation among the centers or staff members are also discussed. 
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Chapter 3 

PACE Girls 


This chapter describes the ways in which PACE recruits girls and determines their eligibility, 
and draws on interviews to offer the perspectives of girls and their parents on why they came to 
PACE. The chapter then presents background characteristics of the entire study sample (both 
the program and control groups) and provides an overview of program group participation in 
PACE. Key findings are as follows: 

• Developing referral relationships is central to PACE’s recruitment efforts; 
schools are the primary referral source. 

• PACE uses a multistep process to assess applicants for suitability for the pro- 
gram. PACE staff members administer a formal assessment and then review 
the girl’s academic and social service needs, as well as her motivation to at- 
tend the program. 

• Girls at PACE have a range of risk factors that qualify them for PACE’s ser- 
vices, including behavior problems, academic underachievement, truancy, 
and a record of suspension or expulsion from school. 

• Girls in the program group enrolled in PACE at a very high rate, and their 
average length of stay at the program was approximately eight months. 


Recruitment 

PACE engages in recruitment efforts in order to maintain an active stream of applicants to the 
program. As PACE’s main funding from the Florida Department of Juvenile Justice (DJJ) and 
the school districts is dependent on the number of girls served, consistently receiving referrals to 
keep program slots filled is central to a PACE center’s financial health. Furthermore, because 
girls may enter and exit the program at any point during the year, recruitment efforts continue 
year-round. Each PACE center had an outreach counselor to manage recruitment during the 
evaluation period. 1 


'The outreach counselor position was specifically created to support the additional recruitment and pro- 
gram intake responsibilities brought on by the study. Before the evaluation, these tasks were largely spread 
among social service staff members. Once study intake activities ended, each center decided whether to 
continue using an outreach counselor or to use a different staffing structure for recruitment efforts. 
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A major part of recruitment for PACE involves building relationships with referral 
sources in the community — conveying what PACE does and what kinds of girls it serves, so 
that partners can identify potential applicants and direct them to the program. PACE’s main 
referral relationships are with schools; half the outreach counselors interviewed noted that 
school guidance counselors or social workers were primary referral sources. Juvenile probation 
officers (JPOs) are another source; while PACE is primarily a prevention program, girls who 
have been or are currently involved with the juvenile justice system can also be referred to 
PACE. JPOs are the source of fewer referrals than schools, however, for at least two reasons 
reported by outreach counselors: Turnover among JPOs results in a constant need to teach new 
officers about PACE, and many girls on probation may not be appropriate for PACE. 

Besides developing these main referral relationships, outreach counselors post flyers or 
leaflets in the community to recruit new girls and often receive new applicants by word of 
mouth from girls who previously attended PACE. 


The Program Application Process 

After receiving a referral or being contacted by a girl’s parent or guardian, the PACE staff 
implements a thorough process to determine whether an applicant is eligible for PACE. Figure 
3.1 illustrates these steps. This application process is largely specified in the policy and proce- 
dures manual. All applicants must first meet a set of basic requirements, including age, sex, 2 and 
having risk factors across a set of domains (family, school, behavior, victimization, and health). 3 

During the application process, the girl and her family receive an orientation to PACE, 
which includes an overview of the program and a tour of the facilities. Staff members conduct a 
fonnal assessment with the girl, called the Initial Needs Assessment (INA). 4 The assessment 
consists of questions structured around the domains given above and helps the staff understand 
the girl’s history and risk factors in greater detail. During the evaluation, an additional step in 


2 Girls who identify as boys are eligible for the program; however, some PACE staff members questioned 
whether they were appropriate for PACE. Staff members also noted that sometimes issues related to a girl’s 
gender/sex identity arose after she enrolled. PACE is reviewing its policies on gender/sex identification to 
address these questions and concerns. 

3 DJJ (a PACE funder) requires that girls exhibit risk factors in at least three domains in order to enroll. 

4 After about half the study sample had been randomly assigned, DJJ required that PACE use a different 
assessment called the Prevention Assessment Tool (PAT). There are minor differences between the PAT and 
the INA. Staff members were trained to use motivational interviewing to collect data for the PAT. 
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Figure 3.1 

Eligibility and Enrollment Process 



Random Assignment 


Girls determined to be eligible for PACE are randomly assigned. 
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the application process provided an overview of the study and asked the girl and her parent or 
guardian to consent to participate in it . 5 

Last, a group of designated staff at the center meets to discuss whether the girl is a good 
fit for PACE and make a final determination of eligibility. The manual offers broad guidance on 
determining eligibility, stating that PACE will make “every attempt” to provide services, but 
that PACE may not be the best fit for all girls referred. Staff members discussed with research- 
ers three main considerations in determining a girl’s fit for the program: 

• Academic needs. Staff members consider whether PACE has the ability to 
serve a girl’s academic needs. Almost all outreach counselors noted that they 
could not serve girls with Individualized Educational Programs that exceed 
the capacity of the center, such as girls with special needs who require a sep- 
arate classroom. Centers also considered how close the girl was to gradua- 
tion; since most PACE centers cannot issue high school diplomas, girls near- 
ing graduation may not be a good fit for the program. 

• Social service needs. Staff members focus on a girl’s behavioral history to 
determine whether they can meet her social service needs. The manual does 
not provide specific guidance on what behavior issues would make a girl a 
poor fit for PACE; it appeared that centers took an individual approach to 
each girl. In reviewing a girl’s behavioral history, staff members try to under- 
stand the cause of behavior issues and determine whether they would be a 
problem at PACE, particularly if there is the possibility of harm to other girls 
at the center. Two counselors also mentioned that PACE was not equipped to 
serve girls with severe mental health problems. Specifically, if the girl falls 
under the Florida Mental Health Act , 6 then by state law, PACE must refer 
her to an institutional facility. There did not seem to be any other finn rules 
on what mental health issues would mean a girl could not be served by 
PACE. 

• Motivation. The manual states that staff members should consider whether 
the girl’s choice to enroll in PACE is voluntary. Nearly all outreach counse- 
lors said that they considered the girl’s motivation in determining eligibility. 


5 Nearly all girls who applied and met both PACE and study eligibility criteria consented to participate in 
the study. In a few cases where PACE believed girls could not be served adequately elsewhere, the study 
participation requirement was waived. 

6 The Florida Mental Health Act, Fla. Stat. § 394.459 (2008), commonly known as the Baker Act, gives 
people with severe mental illnesses the right to emergency mental health services and temporary detention 
(University of Florida Health 2017). 
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Counselors suggested that if girls are not motivated, it would be difficult for 
them to succeed at PACE, but they indicated that they had a low threshold 
for what qualified as motivation. Many described motivation simply as a 
girl’s willingness to attend the program or acknowledgment that she needed 
change in her life, as opposed to showing overt enthusiasm for the program. 

In fact, many PACE girls reported that they were not originally interested in 
coming to PACE. Several counselors noted that they worked with resistant 
girls to persuade them of the value of the program; one explained that often 
such resistance emerged when a girl was told by her parent to go to PACE, 
rather than deciding for herself. Overall, while staff members seemed to view 
motivation as an ideal quality in an applicant, low motivation did not appear 
to be a reason to turn someone away. 

As noted, the manual offered limited guidance on determining eligibility. In December 
2014, after the majority of implementation research visits took place, PACE headquarters 
circulated more specific guidance about eligibility determination and factors to be considered 
during intake. The overall message was one of inclusivity, stressing that turning a girl away 
should be a “rare and serious occurrence.” 

During the study period, as reflected in Figure 3.1, girls were randomly assigned after 
staff members followed all steps in the eligibility determination process described above and 
deemed them eligible for PACE. In most cases during the study period, girls assigned to the 
program group were invited to enroll at the center immediately; if all spaces were filled at the 
center, program group girls were placed on a waiting list and invited to enroll as soon as a spot 
opened up. Girls assigned to the control group were provided a referral list of other options in 
the community. The list was often tailored or customized based on the staffs determination of 
what other programs would be the best fit for the girl based on their assessment of her during 
the intake process. 


Why Girls Come to PACE 

Interviews and focus groups with PACE girls provided insight into the reasons they came to the 
program. Most girls reported that they had attended a public school before coming to PACE and 
had been having some sort of trouble at school. The most prevalent issues included behavior 
problems (such as disrespecting teachers, skipping classes, or not following the rules at home), 
failing classes, truancy, expulsion, and suspension. Girls also cited mental health issues, drug 
and alcohol use or abuse, negative peer influence, and bullying as factors in their arrival at 
PACE. 
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In one interview, a girl discussed her experience with drug use and negative peer influ- 
ence before PACE and how it affected her progress in school. She said she was spending time 
with the wrong group in high school; her best friend was doing drugs and pressured her to 
smoke marijuana, and the more she smoked, the more her grades dropped. She also struggled 
with grief after two of her friends died. 

Many girls said that they did not want to attend PACE initially; some girls did not want 
to go to an all-girls school, and others reported that their mothers or other family members made 
them go to PACE. At the same time, many girls had seen potential benefits to attending PACE. 
Girls had looked forward to the opportunity to catch up academically, to the access to counse- 
lors, and to a caring staff and community. Some of these sentiments are illustrated in excerpts 
from individual interviews with the girls. In one interview, a girl described what she found most 
appealing when she first visited PACE — a place where she felt welcomed: “It is actually going 
to feel like a school. The other ones [programs] didn’t really feel like a school. It just felt like, 
this is where bad people come when they get in trouble. Here it’s not all about bad people. 
Some people just come to get their school work done.” One girl said that she wanted to come to 
PACE because of the program’s close-knit community: “When I first came here I was seeing 
how everyone was so close and how it was like a big family and it made me think of my own 
family, so I just like, fell in love with it.” Another girl was especially drawn to the idea of 
having her own counselor, since at her previous school it was very difficult to meet with the 
counselor. She explained, “ft was hard having nobody to talk to.” 


Background Characteristics of the Study Sample 

Tables 3.1 and 3.2 present demographic and risk characteristics of the study sample at the time 
the girls applied to PACE. 7 This sample closely resembled the types of girls PACE serves more 
generally. 8 Additionally, the risk factors presented are consistent with girls’ self-reports to the 
implementation research team about the issues that brought them to PACE. 

As shown in Table 3.1, the majority of girls in the study (82 percent) were 13 to 16 
years old. Most came from low-income families (77 percent). About half the sample lived in 
single-parent households, and just over one-third came from two-parent households. Forty-two 
percent of sample members came from families that had had some type of involvement with the 
Department of Children and Families, which handles cases of abuse and neglect. 


7 See Appendix Tables A.l and A.2 for characteristics by research group. 

8 In fiscal year 2015, PACE served a total of 2,130 girls. As context for some of the figures cited in the rest 
of this section, among all girls served in fiscal year 2015, 74 percent came from low-income families, 74 
percent had failed a class recently, 29 percent exhibited runaway behavior, and 28 percent had been arrested in 
the past. 
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Table 3.1 


Characteristics of Sample Members at Baseline 


Characteristic (%) 

Full 

Sample 

Age 

11-12 

8.5 

13-14 

32.5 

15-16 

49.5 

17 or older 

9.5 

Race/ethnicity 

Black, non-Hispanic 

45.1 

Hispanic 3 

16.0 

White, non-Hispanic 

38.1 

Other 

0.8 

School level at time of referral to PACE 

6th grade b 

8.8 

7th-8th grade 

37.2 

9th- 10th grade 

45.3 

1 1th- 12th grade 

8.7 

English is second language 

2.1 

Qualifies for special education or ESE L 

11.2 

People participant lives with 

Two parents 

34.8 

Single parent 

51.8 

Relative 

10.6 

Other d 

2.8 

Family income 6 

$28,050 or lower 

41.2 

$28,051-$44,900 

35.5 

Above $44,900 

23.3 

Family has had case with the 

f 

Florida Department of Children and Families 

42.2 

Sample size 

1,134 


(continued) 
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Table 3.1 (continued) 


SOURCE: MDRC calculations based on data from the PACE management 
information system. 

NOTES: a Sample members are coded as Hispanic if they answered "yes" to 
Hispanic ethnicity. 

b This category includes sample members who were in fifth grade at the time of 
referral. 

C PACE uses the Florida Department of Education definition of Exceptional 
Student Education (ESE), referring to programs for students with disabilities and 
gifted programs. 

d " Other" includes nonrelative or foster care. 

e The U.S. Department of Housing and Urban Development's guidelines were 
used in the data collection process to determine which income range the 
participant's family fell into. Since these figures could vary by county and 
household size, the ranges presented here correspond to statewide income limits 
for low income and very low income for a four-person household in Florida in 
fiscal year 2014. 

f This measure is a reflection of what sample members self-reported; it is not 
according to their parents or guardians. 


These data indicate that sample members struggled in school before coming to PACE. 
As shown in Table 3.2, at the time of random assignment, three-quarters of girls in the study 
had failed a class recently, and just over half had been held back at least once. Significant 
portions of the study sample also had low school attendance (42 percent) or had been recently 
suspended or expelled (40 percent). 9 These rates of academic risk are considerably higher than 
the rates among all students in the counties where PACE centers are located; in the 2013-2014 
school year, only 5 percent of students were held back, 10 percent had low school attendance, 
and 15 percent were expelled or suspended. 10 

Girls in the study sample face a number of health and safety risk factors. As shown in 
Table 3.2, a significant portion of the sample, 44 percent, had been or were currently sexually 
active. This figure is similar to the percentage of high school girls in Florida in 2013 who had 
ever had sexual intercourse (40 percent), even though the PACE study sample consists of both 


9 Low school attendance is defined as more than 15 absences in the past three months. 

10 These data were not available by grade. They were calculated from Florida Department of Education 
data as the number of students absent 2 1 or more days over total enrollment for the counties where participat- 
ing PACE centers are located. 
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Table 3.2 


Risk Factors of Sample Members at Baseline 


Characteristic (%) 

Full 

Sample 

School engagement 


Recently expelled or suspended 1 * 

39.6 

Currently enrolled in school 

73.1 

Skipped school at least 3 times in past 2 months 

34.7 

Had more than 1 5 absences in past 3 months 

41.7 

Held back at least once 

51.8 

Failed 1 or more classes in past 6 months 

76.6 

Has a learning disability 

29.6 

Attention deficit disorder 

19.4 

Dyslexia 

1.5 

Other learning disability 

8.8 

Delinauencv 


Ever been arrested 1 * 

27.7 

Ever stolen from family, home, or neighbors 

16.7 

Ever been on probation 

12.6 

Currently on probation 

10.1 

Has family member with criminal history 1 * 

64.1 

Has friends with delinquent record or who engage in 


delinquent behavior 

49.8 

Health and safety 


Currently using tobacco* 1 

9.9 

Currently using drugs or alcohol 1- 

14.8 

Ever sexually active 

44.1 

Currently pregnant 

1.4 


(continued) 
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Table 3.2 (continued) 


Characteristic (%) 

Full 

Sample 

Ever run away from home 

27.6 

Ever had thoughts about harming/killing herself 

39.3 

Abused/neglected 1 

38.1 

Neglected 

8.9 

Physically abused 

15.9 

Sexually abused 

15.1 

Emotionally abused 

21.7 

Sample size 

1,134 


SOURCE: MDRC calculations based on data from the PACE management 
information system. 

NOTES: Certain characteristics listed here were captured in two different ways 
during the random assignment period, as noted below. 

a For approximately half of the sample, this was defined as being currently 
expelled or suspended. For the other half of the sample, this referred to one or 
more expulsions or suspensions in the most recent school term. 

b ln the juvenile justice system, people are not technically "arrested"; the 
terminology used is either "incurred a charge" or "referred." 

c For approximately half of the sample, this measure referred to a criminal 
record (including imprisonment, probation, parole, and house arrest) for a 
parent, guardian, or sibling of the sample member. For the other half of the 
sample, "family" included other members of the household as well. 

d For approximately half of the sample, this was defined as having used 
tobacco three or more times in the past 30 days. For the other half of the sample, 
this was defined as currently using tobacco. 

e For approximately half of the sample, this was defined as having used drugs 
or alcohol three or more times in past 30 days. For the other half of the sample 
this was defined as current drug and or alcohol use. 

f For approximately half of the sample, this measure referred only to 
documented instances of abuse or neglect. For the other half of the sample, the 
measure also included suspected incidents of abuse. 


middle and high school girls. 11 Approximately 28 percent of sample members had ever run 
away from home. In line with some of the issues girls reported during interviews, 38 percent of 


u Florida-wide percentages refer to female-only Florida responses to the Youth Risk Behavior Surveil- 
lance Survey in 2013 (Centers for Disease Control and Prevention 2016). 
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girls in the study had experienced at least one type of abuse (physical, sexual, or emotional) or 
neglect, and 39 percent reported having thoughts about harming or killing themselves. The high 
incidence of these experiences presents a need for what is known as trauma-informed care, as 
discussed in Chapters 4 and 6. 

About 28 percent of sample members reported having been arrested before coming to 
PACE. Unsurprisingly, given PACE’s target population, this percentage is much higher than the 
official arrest rate for the broader population. In 2014, only 1 percent of the Florida female 
population ages 10 to 17 had ever been arrested. Ten percent of the study sample were on 
probation at the time of study enrollment, while less than 1 percent of the Florida female youth 
population had been on probation in fiscal year 2013-2014. 12 In terms of other delinquency risk 
factors faced, a majority of the sample had a family member with a criminal history (64 per- 
cent). 


Program Group Participation 

Most girls assigned to the program group enrolled at PACE (90 percent) and did so fairly 
quickly — within about two weeks. Table 3.3 provides measures of participation at PACE 
among the program group. 13 According to PACE’s program data, among those who enrolled at 
PACE, about three-quarters stayed for longer than 90 days. 14 On average, girls were enrolled for 
just under eight months, but some girls in the study sample stayed much longer, up to 33 
months. 


In the 12-month follow-up survey, the percentage of girls who reported having attended 
PACE matched the program’s administrative records. 15 As Table 3.3 shows, among the few 
who did not attend, the most commonly reported reason was the decision to enroll in another 
school or program instead (36 percent). Almost one-quarter reported not coming to PACE 
because they lacked transportation; this was also an issue regularly noted by center staff 
members in interviews with the implementation team. Nonattendees also reported that they did 
not enroll in PACE because they decided they did not like the program (23 percent) based on 
their introduction to it during the application process. This disinclination could involve not 
wanting to go to an all-girls school or other components of PACE that are unlike the traditional 
public school. 


12 Data derived from Florida Department of Juvenile Justice (2016). 

13 These figures are based on data from the PACE management information system and a partial sample 
from the 12-month follow-up survey. The final report will update these figures for the full sample. 

14 This measure captures only a girl’s first enrollment at PACE. 

15 Only 31 program group girls among the survey sample respondents thus far had not attended PACE for 
at least one day. 
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Table 3.3 


Program Participation (Program Group Only) 


Service 

Program Group 

Enrolled in PACE day program (%) 

90.3 

Among girls who enrolled in PACE 

Time from random assignment to enrollment (days) 

19.4 

Average daily attendance (%) 

72.0 

Length of stay d (months) 

7.9 

Length of stay a (%) 

Less than 30 days 

7.3 

Between 30 and 90 days 

18.3 

90 days or more 

74.4 

Among girls who did not enroll in PACE h 

Reasons for not coming to PACE (%) 

Enrolled in another school or program 

35.5 

Did not have transportation 

22.6 

Did not like the program 

22.6 

Moved 

12.9 

Family member became ill 

9.7 

Parent or guardian did not like the program 

9.7 

Other c 

25.8 

Sample size 

679 


SOURCES: MDRC calculations based on data from the PACE 
management information system and the 12-month follow-up survey. 


NOTES: “This measure represents only a participant's initial stay at 
PACE. 

b Calculations are based on responses to the 12-month follow-up survey 
and include girls randomly assigned through March 2015 (N = 31). 

c The percentage in the "other" category appears high, but it represents 
only eight girls. 
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Chapter 4 


Gender-Responsive Programming 
and PACE’s Program Environment 


This chapter provides a description of the key components of PACE’s intended program 
environment and approach to gender-responsive programming and explains how staff members 
put these concepts into action . 1 The chapter presents the following main findings: 

• PACE’s program environment is its foundation for providing gender- 
responsive programming. The key components of this environment are a fo- 
cus on safety, relationships between staff members and girls, recognizing and 
promoting girls’ individual strengths, and responding to the needs of girls 
who have experienced trauma. 

• Ensuring that programs foster a common culture across multiple locations is 
a challenging task. PACE has successfully done so, with staff members de- 
scribing a similar program environment regardless of their role or location. 
Interviews with girls and parents corroborate this: They experienced a safe 
and welcoming program environment similar to what the staff intended. 

• Findings from the Organizational Social Context tool, a measure of organiza- 
tional culture and climate, were consistent with staff descriptions from inter- 
views. Results indicate a culture and climate that gave priority to meeting the 
needs of the client; high morale among the staff; and indications that staff 
members felt a limited ability to change organizational processes, likely re- 
flecting PACE’s fonnalized approach to model implementation. 


Gender-Responsive Programming 

Gender-responsive programming grew out of the recognition that girls involved in the juvenile 
and criminal justice systems have particular needs. Girls’ pathways into the justice system are 
typically different from those of boys. Girls in the juvenile justice system are more likely than 
boys to have experienced sexual violence, extreme family conflict, and child maltreatment . 2 


'in this chapter, program environment refers to the internal environment of the PACE centers, specifically 
the environment that girls, families, and staff members experience while at the centers. This chapter does not 
cover the external program environment in which the center operates. 

2 Bright and Jonson-Reid (2008, 2010). 
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Because of these different experiences, girls and boys need different services, but most preven- 
tion programs have been designed for boys. 3 Gender-responsive programs were developed to 
provide an approach that is tailored for girls. However, the field is still developing, and evidence 
on the effectiveness of gender-responsive programming is sparse. 4 

A key research question for this evaluation was to understand how PACE puts the theo- 
ry of gender-responsive programming into action. There is no single definition of gender- 
responsive programming, but Table 4.1 presents commonly cited principles and shows how the 
components of PACE’s model correspond to these principles. The components fall into four 
main areas: 


• Program environment. Providing girls with a program environment that is 
safe and that integrates relational, strengths-based, and trauma-informed ap- 
proaches in all aspects of program delivery (detailed in this chapter). 

• Assessment. Employing a comprehensive assessment process to understand 
a girl’s history, strengths, and risk factors for delinquency (detailed in Chap- 
ter 3). 

• Life s kil ls. Providing girls with a gender-specific life skills curriculum that 
focuses on issues particular to a women’s health, supports career readiness, 
and offers connections to community (detailed in Chapter 5). 

• Parental engagement. Engaging the family in a girl’s care at PACE (de- 
tailed in Chapter 6). 


Program Environment 

Program environment is often a focus in youth programs because it is viewed by many re- 
searchers and practitioners as a key aspect of service delivery and as connected with youth 
outcomes. Aspects of program environment include how staff members interact with partici- 
pants, rules governing behavior, the program’s physical space (decorations and security fea- 
tures), and how participants are involved in leadership within the program. Program environ- 
ment is also seen as central to the implementation of gender-responsive programming. 5 For 
these reasons, MDRC’s implementation research examined PACE’s program environment in 
depth. 


3 Zahn, Hawkins, Chiancone, and Whitworth (2008). 

4 Hubbard and Matthews (2008). 

5 Covington and Bloom (2006). 
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Table 4.1 


Gender-Responsive Programming Principles and 
PACE Program Components 


Category 

Principle of 

Gender-Responsive 

Programming 

PACE Program Component 

Program 

environment 

Safety 

PACE provides secure facilities, behavior management, and a 
program culture that is intended to be safe from bullying and 
trauma triggers. 


Focus on high-quality 
relationships 

Staff members focus on building positive and supportive 
relationships with the girls. Care is informed by the other key 
relationships in a girl’s life, including family relationships. 


Strengths-based 

approach 

Staff members are trained to recognize a girl’s assets and orient 
care toward building strengths rather than focusing on deficits. 


T rauma-informed 
approach 

Staff members are trained to recognize the symptoms of trauma 
and to understand how trauma can affect a girl’s behavior. Staff 
members use knowledge of a girl’s trauma history to inform care. 

Assessment 

Holistic approach to 
treatment 

PACE implements a comprehensive assessment process to 
understand a girl’s risk factors and protective factors across five 
domains: family, school, behavior, victimization, and health. 

Life skills 

Education about 
women’s health 

The Spirited Girls! life skills curriculum educates girls about 
healthy relationships and general and reproductive health. Staff 
members work with girls to address specific women’s health 
needs. 


Educational and 
vocational opportunities 

Academic services provide girls with an opportunity to catch up to 
grade level by providing individual support in small classes. 

Career exploration is provided in Spirited Girls! classes or through 
separate career classes. Staff members provide individual support 
on career planning. 


Connections to the 
community 

Volunteer service provides girls with the opportunity to connect 
with the community in a positive way. 

Parental 

engagement 

Emphasis on family 

Staff members engage a girl’s family in her care through regular 
updates on her progress and by seeking to address needs within 
the family when possible. Staff members use an awareness of 
each girl’s family dynamics to inform her care. 


SOURCES: Developed from Kerig and Schindler (2013) and interviews with PACE staff members. 
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PACE staff members at all levels and across all participating centers were asked to de- 
scribe the environment they intended to create for girls. Staff members were also asked how 
they implemented a gender-responsive approach in their day-to-day work. In all, 112 staff 
members across 14 centers were asked questions about the environment and gender-responsive 
programming. Responses to these questions focused on the program environment principles 
listed in Table 4.1: safety, high-quality relationships, a strengths-based approach, and a trauma- 
infonned approach. These components are also more generally cited as important to effective 
programs for young people. 6 Staff responses related to each of these topics are summarized 
below. The following sections also describe the perspectives of girls and parents based on 
interviews and the 12-month survey of girls. 

Safety 

In line with the principles of both youth and gender-responsive programming, staff 
members first and foremost emphasized that the environment should be safe from both physical 
and emotional threats. One senior manager at PACE described the importance of safety at 
PACE and its impact on girls: “Eve never seen anything like the level of . . . emotional and 
physical safety that’s present in a center. And you can see it in the girls in the way they act. And 
the way they show up as girls, as kids, instead of with all those extra sort of layers.” 

PACE employed several strategies to support physical safety. To control who enters 
and leaves the facilities, all PACE centers are secured with locked external doors that require 
visitors to be buzzed in and registered at a front desk. Girls were required to store their personal 
belongings in a designated area upon entering to prevent them from bringing in unsafe or 
distracting items. Rooms that were not in use were locked to prevent girls from accessing areas 
that were not supervised by staff. The main physical threat mentioned was fighting between 
girls enrolled in the program. To prevent this, the staff practiced “sight and sound” supervision, 
which meant ensuring that all girls were within hearing or sight of a staff member at all times. 
Staff members also emphasized the importance of knowing the girls and their current situations 
as a way to maintain safety within the center. Staff members would pay closer attention to girls 
they knew were in crisis. Daily communication, such as morning meetings, were used to update 
staff members on emerging issues with or between girls in the program. 

PACE’s discipline policy promotes safety. Many staff members said the response to a 
behavior issue was tailored to each girl and her circumstances, but staff members would 
consider the safety of all girls in the center when deciding on consequences. Some centers had 
zero tolerance policies around fighting, and girls who fought would be expelled (“transitioned”). 
Nonphysical infractions, such as bullying or being insubordinate to a teacher, could result in a 

6 Catalano et al. (2004); Ko et al. (2008). 
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suspension, either in school or out of school (known as “days of reflection”). The PACE policy 
and procedures manual provides guidance on discipline policies, outlining the approaches that 
centers should take and prohibiting discipline techniques such as “use of isolation.” Some 
centers were required to adopt the local school district’s student code of conduct and zero- 
tolerance policies. 

Preventing bullying was a particular focus, and centers provided antibullying education 
through Spirited Girls! (life skills) classes or special events, such as schoolwide assemblies 
dedicated to celebrating diversity and promoting a “sisterhood” culture within PACE. Centers 
structured responsibility for discipline in different ways. At about half the centers, management 
staff members rather than counselors disciplined girls so as to preserve the therapeutic relation- 
ship between counselors and girls. At the other half, counselors did have a role in discipline, and 
managers believed it was beneficial to have a counselor help a girl process the issues that 
caused the behavior. 

Most girls and parents described PACE as a safe place. Parents were also overwhelm- 
ingly positive about the environment they encountered at PACE; most described it as a safe 
environment and said incidents of bullying were dealt with appropriately. In describing the safe 
aspects of the environment in interviews, girls said that staff members monitored the center 
closely, and girls who were a threat to the safety of the school would be expelled. One girl said: 
“If the staff finds out there’s going to be a fight they are on top of it and end it right then and 
there.” Girls and parents liked that the buildings were secure and that PACE limited access by 
outsiders. Most girls did not think bullying was a problem, noting that staff members were on 
the lookout for bullying and would address it immediately. Not all girls felt safe, however: 15 
percent of girls in the follow-up survey said they felt concerned about safety while at PACE. 

Girls did say that being in an all-girls environment could lead to “drama,” which was 
described as distinct from bullying, referring instead to minor issues between girls, such as two 
girls having the same romantic interest. Though girls in interviews could be cavalier about the 
“drama,” the follow-up survey of girls suggests that it could become a more significant issue for 
some; of girls who had left PACE, nearly a third of them reported that not liking or getting 
along with other girls in the program was a contributing factor. 

High-Quality Relationships 

After emphasizing that the centers should have a foundation of safety, staff members 
most often said that the enviromnent should provide nurturing relationships for the girls. 
Common tenns used by staff members included “loving,” “family-like,” “positive,” and 
“supportive.” One counselor said: “It’s a loving enviromnent. When girls walk in the door, they 
know that PACE staff care for them.” Staff members described their relationships with girls as 
central to implementing a gender-responsive approach, and their answers aligned with gender- 


37 



responsive theory about the relational needs of girls. This theory holds that relationships are 
central to girls’ lives and to their self-esteem. 7 In the words of one female staff member: “Girls 
are relational, everything that happens in our lives is based on a relationship, whether it’s good, 
bad, or indifferent.” Staff members reported that most girls at PACE have had harmful relation- 
ships in their lives, and staff members attempted to counteract that by building positive, affirm- 
ing relationships. 

Staff members described a number of strategies to create nurturing relationships, most 
of which focused on the staffs role and interaction with the girls. They emphasized the im- 
portance of knowing each of the girls and their backgrounds, both to help the girls feel wel- 
comed and to try to avoid actions or activities that might be a trigger, or reminder, of a girl’s 
prior trauma. Staff members described being friendly with all the girls at the center, not just 
those who were in their classes or on their caseloads. Research team members observed this 
strategy in practice during site visits; staff members would greet all the girls who walked by 
them, and it appeared that many girls felt comfortable dropping in on the executive director or 
other members of the management staff. 

Noting that girls might have had limited opportunities to see what a healthy relationship 
looks like, staff members reported modeling such relationships both with the girls and with each 
other: For example, they would interact with girls in the way that they wanted girls to interact 
with them, being positive and upbeat in their exchanges. Staff members would also encourage 
other positive relationships in girls’ lives. Within the center, these efforts could involve peer 
mediation between girls who were not getting along, to address the source of conflict and help 
girls leam how to prevent further problems; in addition, counselors would work with girls on 
ways to promote healthy relationships with their families and romantic partners. 

Girls described positive relationships with the staff members at PACE. Most girls inter- 
viewed said the staff members were the best thing about PACE. They described them as 
“family-like” and “loving,” similar to how the staff described the intended environment. In the 
words of one girl, there was “unsolicited, genuine care between the staff and the students.” 
Nearly all girls interviewed said there was at least one PACE staff member with whom they 
could discuss personal issues; this was most commonly a girl’s counselor but also included 
others at the center. Positive feelings toward PACE staff members were not universal, however: 
In the follow-up survey, 1 8 percent of girls in the program group said they felt misunderstood 
by the PACE staff. 

Though most girls interviewed had positive feelings about the program culture, some 
expressed dislike or discomfort with the way staff members related to them. These girls tended 


7 Gilligan (1982). 
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to say they felt “babied” by the staff, giving the example of teachers or counselors talking down 
to them or treating them as if they were younger than they were. One girl felt uncomfortable at 
PACE because, in such a small school, she could not “hide” in the way she could at her larger 
district school. 

Nearly all parents interviewed said that they felt comfortable at their daughter’s PACE 
center, many because the staff made them feel welcomed. Parents said they could get to know 
the staff at the center because of its small size compared with public school. Describing the 
environment at PACE, one parent said: “I had a smile on my face when I first went into that 
school.” 


A Strengths-Based Approach 

Staff members also talked about the importance of building girls’ self-confidence 
through a strengths-based approach. The strengths-based approach, which originated in the 
social work field, focuses on identifying and building on a client’s strengths and assets to help 
her achieve her goals. Staff members noted that girls may not get much encouragement from 
their families and may not be used to hearing positive things about themselves. This emphasis 
on strengths was implemented formally through PACE’s rewards system and informally 
through the day-to-day interactions between the staff and the girls. 

The PACE manual requires each center to implement a rewards system to recognize 
and encourage positive behaviors in the girls. The specifics of the rewards program varied by 
center, but in general girls could advance to reward “levels” by completing specified milestones, 
such as maintaining a certain attendance rate. Rewards were either tangible (such as clothes or 
make-up) or privileges (a lunch date with a staff member or allowances in the dress code). 
Some centers also had a points system that was separate from the level system and offered girls 
ways to earn rewards by the day or week. The points systems offered girls more immediate 
recognition for positive behavior, while the level system focused on longer-tenn goals. Al- 
though each center offered a rewards system, interviews with staff members and girls revealed 
that girls’ involvement varied. Staff members strived to create a system that would appeal to 
diverse interests, but not every girl wanted to participate. In addition to the system, staff mem- 
bers would be on the lookout for positive behavior and used “gotchas” (where a staff member 
would provide immediate and spontaneous recognition) to reward a girl for doing something 
positive. 

Counselors also took a strengths-based approach in their one-on-one sessions with girls. 
In these sessions, counselors helped girls recognize their strengths and referred back to those 
strengths when helping girls develop strategies to address the challenges in their lives. Counse- 
lors also tried to help parents see their daughters’ strengths by noting them in parent-counselor 
meetings along with areas for improvement. Some staff members focused on “female ernpow- 
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erment” themes when talking about their strengths-based approach, such as helping girls 
understand that their job opportunities in life were not limited by their gender. 

In follow-up interviews about PACE’s impact on their lives, girls most commonly said 
that they learned to recognize positive things about themselves while at PACE: for example, 
helping others with school work, being a good listener, and being a good mother. More broadly, 
many girls said they had higher self-esteem after attending PACE. Reflecting on her time there, 
one girl said she learned that “I was worth helping, worth giving a second chance, and worth 
realizing my potential.” 

A Trauma-Informed Approach 

A trauma-informed approach describes an organization’s response to the needs of those 
who have experienced trauma. A trauma-informed approach refers to a component of an 
organization’s culture; trauma-specific interventions refer to clinical approaches used to 
promote a client’s recovery from trauma. Commonly cited components of a trauma-informed 
approach include a focus on safety, transparency in organizational operations to build trust, a 
focus on collaboration and relationships, a strengths-based approach, and recognition of 
cultural, historical, and gender issues . 8 PACE’s implementation of several of these key compo- 
nents were described in earlier sections of this report. The current section describes how PACE 
supports the implementation of a trauma-informed approach through training and ongoing 
professional development. 

All PACE staff members who interact with girls go through trauma training when they 
first start at PACE and annually thereafter. Staff members are trained to understand the impact 
of trauma on a person’s life, to recognize the symptoms of trauma, and to interact with girls in a 
way that avoids inadvertently contributing to their trauma and supports their healing. Staff 
interviews indicate that PACE has been successful in this training. Staff members described 
efforts to ensure that PACE was a safe place emotionally and that its environment would not 
trigger the girls’ prior traumas. For example, staff members would ask permission before 
touching a girl or warn girls if the content of a movie or book being read in class dealt with a 
difficult subject. 

Another aspect of PACE’s trauma-informed approach involves understanding the im- 
pact of trauma on a girl’s behavior. For example, staff members said, if a girl was being 
disruptive in class, they would try to recognize how her experience might be affecting her as 
opposed to taking a punitive approach. One staff member put it this way: “If [the girls] had a 
trauma that caused them to have high anxiety about social situations, then that’s going to come 


8 Substance Abuse and Mental Health Service Administration (2014). 
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up in everything that they do every day. You have to keep that in mind that she has had this 
horrible experience.” 


Fostering a Gender-Responsive Program Culture 

Fostering a similar program culture across multiple program locations is challenging, as each 
center operates in its particular external enviromnent and staff members are not shared across 
centers. PACE promotes its program culture primarily through a set of values and guiding 
principles, shown in Box 4.1. These values and guiding principles were woven into many 
aspects of program operations. They were prominently displayed at the centers, and staff 
members referred to them when they talked to girls and sometimes when they talked to each 
other. New staff members received training on the values and guiding principles, as well as 
training on staff role modeling, gender-responsive programming, strengths-based behavior 
management, and trauma-informed care. 

The research found that each PACE center reflected a program culture that was con- 
sistent with the overall PACE values and guiding principles. Staff responses to the interview 
questions about program environment showed strong consistency in descriptions of the ideal 
PACE enviromnent and the strategies to achieve it across roles and centers. Moreover, most 
staff members at different levels in the centers — executive directors, middle management, and 
counselors — gave similar answers to questions about how they incorporate a gender- 
responsive approach into their work. 

Teachers were the only group that gave less consistent responses. Although many 
teachers provided answers that demonstrated use of a gender-responsive approach, about 40 
percent of teachers interviewed either had difficulty describing how they used a gender- 
responsive approach or said that they do not use such an approach in their work. A few teachers 
admitted to not fully subscribing to the gender-responsive approach. Others would offer 
descriptions of how they incorporated such programming into their classrooms, but the exam- 
ples they provided did not in fact reflect gender-responsive principles. For example, in describ- 
ing the primary way that they were gender responsive, one teacher said that girls should be 
taught to be “civil” while another said she focused on the decorations in the classroom. The 
lower level of knowledge among the teachers about gender-responsive approaches was associ- 
ated with shorter tenures at PACE. The majority of teachers who had difficulty describing their 
use of a gender-responsive approach had been at PACE for less than a year; half of them had 
been at PACE for less than six months. 

Broadly, however, the PACE centers in this study shared a common program culture, 
even though they were operating in different local contexts. The uniformity of staff responses 
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Box 4.1 


PACE Values and Guiding Principles 

Honor the Female Spirit. We value and promote the female perspective by respecting its 
distinct needs, creating safe gender-responsive environments, and celebrating the female 
experience. 

Focus on Strengths. We look to identify strengths in our girls, their families, our staff and 
supporters. Using these strengths as our foundation, we build strong, confident, productive 
community participants. 

Act with Integrity and Positive Intent. We believe that all actions and decisions must be 
guided by the highest ethical principles, respecting the uniqueness of all involved and honoring 
the differences. 

Embrace Growth and Change. We believe that everyone is capable of remarkable growth, 
and only by encouraging change can individuals, organizations and society reach their full 
potential. 

Value the Wisdom of Time. We understand that patience can be as powerful as immediate 
action, and each has its place. We value the discernment required for their effective use. 

Exhibit Courage. We think courage is essential in making a difference, enabling us to speak 
for those who cannot, take risks to do what is right, deliver just and fair consequence and be 
accountable for our actions. 

Seek Excellence. We strive for excellence in all we accomplish by holding true to our mission 
while consistently meeting high standards of performance, reflecting critically upon our 
accomplishments, seeking innovative solutions, and believing all things are possible. 

Create Partnerships. We believe in developing effective partnerships and long-term relation- 
ships, by listening to our staff, our girls, their families and our communities, incorporating their 
input and involving them in our decision-making. 

Invest in the Future. We place our faith in the long-term growth and development of our 
girls, staff, agency and communities, believing it is the best strategy for creating results that 
have lasting impact. 

SOURCE: PACE policy and procedures manual. 


(with the exception of some teachers, noted above) is related to the training staff members 
receive and the structured approach to model implementation. The consistency of staff descrip- 
tions of the program culture may also be a reflection of the consistency of messages from PACE 
leadership: The chief executive officer and executive directors used a common language to 
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describe the intended environment and gender-responsive programming, and the language used 
by direct service staff members mirrored that of their managers. The finding that teachers with 
shorter tenures had a difficult time articulating PACE gender-responsive approaches indicates 
that staff turnover, and the need to train new staff members on the program environment, can 
cause barriers to implementing the program environment as intended. 

Measuring Program Environment 

Measuring the environment of a program is often a difficult and subjective task — pro- 
grams sometimes refer to their program environment as the “secret sauce” that no one knows 
how exactly to measure or replicate. This evaluation used the Organizational Social Context 
(OSC) tool as a measure of PACE’s program environment. 9 Social context, or an organization’s 
culture and climate, is thought to be a factor in human service program implementation, 
associated with staff adoption of organizational practices, fidelity, staff turnover, and relation- 
ships between the staff and clients. 10 This study uses the OSC as one measure of the social 
context at PACE centers. As described in this section, the OSC results are consistent with the 
research team’s findings from interviews with PACE staff members. 

The OSC is designed to measure organizational culture, organizational climate, and 
worker morale, discussed further below. These constructs are assessed through a 105-item 
survey that asks staff members to rate on a five-point scale how often each statement applied to 
their center experiences. The scale ranges from “never” to “always.” Culture and climate are 
each made up of subscales, described in Box 4.2. Morale, also shown in Box 4.2, measures 
individual job satisfaction and commitment to the center. 11 OSC scores are calculated for each 
organizational unit as t-scores, or standardized scores, where the mean is 50 and the standard 
deviation is 10. 12 The mean and standard deviation were established from a national sample of 
100 mental health organizations across the nation; the respondents in the national sample were 
all mental health clinicians providing services to children and adults. 13 


9 The OSC was administered to all direct service staff members. 

10 Glisson et al. (2007); Hemmelgam and Glisson (1998); Glisson and Green (2006). 

1 Scores for culture and climate subscales are calculated at the organizational level, which in this case was 
each PACE center. Morale scores are first calculated at the individual level and then averaged for each center. 

12 Standardized scores ensure that the scores are normally distributed, with half the scores falling below the 
mean and half falling above the mean. As an example, a center that had a score of 60 on proficiency would fall 
above the scores of more than 84 percent of the agencies in the sample. 

13 The sample was derived from a subset of participating clinics in the National Survey of Child and Ado- 
lescent Well-Being. Across the participating clinics, 1,154 clinicians took the OSC. See Glisson, Green, and 
Williams (2012). 
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Box 4.2 


Organizational Social Context Subscales 

Culture 

Proficiency measures competency and responsiveness. Competency is the degree of emphasis 
an organization places on job knowledge and expectations of excellence from employees. 
Responsiveness measures the extent to which the staff is expected to place the well-being of 
clients first and meet their unique needs. One example of a survey statement that measures 
proficiency is “Members of my center are expected to be responsive to the needs of each 
client.” 

Rigidity measures centralization (the extent to which power is diffused within the organization) 
and formalization (the extent to which procedures guide work-related interactions). In rigid 
environments, the staff has less ability to exercise discretion and limited power to influence 
key management decisions. One example of a survey statement that measures rigidity is “The 
same steps must be followed in processing every piece of work.” 

Resistance measures apathy toward change and suppression of change. Apathy is the extent to 
which the staff is resigned to the status quo versus interested in actively seeking new ways of 
providing service. Suppression looks at how attempts at change are handled within the organi- 
zation. One example of a survey statement that measures resistance is “Members of my center 
are expected to not make waves.” 

Climate 

Engagement looks at the degree to which the staff feels connected and involved with clients, as 
well as the degree to which staff members feel that they are effective in their roles. A survey 
item that measures engagement is “1 have accomplished many worthwhile things in this job.” 

Functionality looks at cooperation among staff members, clarity of staff roles, and opportunity 
for growth and advancement. Two survey items that measure functionality are “My job 
responsibilities are clearly defined” and “PACE provides numerous opportunities to advance if 
you work for it.” 

Stress measures whether staff members feel overloaded, overwhelmed, and emotionally 
exhausted by their jobs. Survey items include “1 feel like I am at the end of my rope” and “The 
amount of work I have to do keeps me from doing a good job.” 

Morale 

Morale measures staff satisfaction with and commitment to their jobs. Survey items include “I 
really care about the fate of this organization.” 

SOURCE: Paraphrased from a report provided to MDRC by the Center for Behavioral Health Research. 
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The scores for PACE centers are shown in Figure 4.1. Each dot represents the t-score 
for a center, and the national mean of 50 is shown as a horizontal line. As shown, nearly every 
center had a t-score above 50 on each subscale. These scores indicate that PACE centers 
exhibited a different OSC profile from that of the national comparison sample, which likely 
reflects, at least in part, the different contexts that PACE programs operate in compared with 
mental health organizations (for example, academic services rather than health care) and the 
different types of roles that staff members at PACE play compared with mental health clini- 
cians. While t-scores tended to be high, the culture and climate subscales show a range of 
scores. Morale scores, by contrast, are clustered. The next paragraphs discuss possible interpre- 
tations of these scores and limitations of their use. 

Organizational climate is defined by the OSC as “the psychological impact of the work 
environment on the individuals who work there.” 14 Compared with a national sample of mental 
health organizations, most PACE centers scored above the mean on the OSC climate subscales: 
engagement, functionality, and stress. Engaged cultures reflect organizations where staff 
members feel personally invested in their jobs, which is how many staff members described 
their work at PACE. Nearly all PACE centers scored high on the OSC’s measure of functionali- 
ty, which indicates that PACE staff members generally feel that there is clarity around individu- 
al staff roles and opportunities for advancement within the organization. Most PACE centers 
scored near or above the mean on stress, indicating that although they were engaged in their 
work and there was clarity around roles, they still found their jobs to be emotionally exhausting 
or overwhelming to some degree. This is not surprising given the needs of the population PACE 
works with. 

Organizational culture is defined by the OSC as “how things are done” and the expecta- 
tions of the organization for its staff. 15 Compared with the national sample of mental health 
organizations, most centers scored above the mean on the culture subscales: proficiency, 
rigidity, and resistance. According to OSC documentation, a high proficiency score is associat- 
ed with organizations where employee excellence and competence are valued and staff mem- 
bers are expected to put the needs of clients first. 16 As described in the previous section, a strong 
“client first” culture emerged from the interviews with staff members, where staff decision- 
making is driven by what is in the best interest of the girls. PACE’s scores on the rigidity and 
resistance subscales indicate that staff members feel they have limited discretion over job tasks 
and limited input into organizational practice. PACE’s manual and implementation support, 


14 Center for Behavioral Health Research (2016). 

15 Center for Behavioral Health Research (2016). 

16 This information is from a report prepared for MDRC by the Center for Behavioral Health Research. 
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Figure 4.1 

Distribution of Organizational Social Context Scores by Center 
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OSC subscales 


SOURCE: PACE evaluation staff survey using the Organizational Social Context (OSC) tool (Center for 
Behavioral Health Research 2016). 

NOTE: OSC scores are calculated as t-scores, or standardized scores, where the mean is 50 and the standard 
deviation is 10. Each dot represents the t-score for a center, and the national mean is shown as a horizontal line. 
Standardized scores were calculated by the OSC developer, the Center for Behavioral Health Research at the 
University of Tennessee at Knoxville. 
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including extensive training and quality assurance audits described in Chapter 2, may be 
reflected in these scores. Policy and procedures are promulgated by PACE headquarters in 
Jacksonville, so most staff members who work with girls would have limited interaction with 
the PACE policymakers. Additionally, many components of the PACE model are highly 
specified in the manual, such as how frequently data should be entered into the management 
information system. The specificity of many aspects of staff responsibilities seems to be 
reflected in the high OSC rigidity subscale score. 

OSC morale scores reflect “employees’ satisfaction with their jobs and their commit- 
ment to the organization.” 17 PACE centers’ scores clustered above the mean, indicating that 
morale at PACE is higher than the average from the national sample of mental health organiza- 
tions. This finding aligns with the personal investment in PACE that staff members described 
during interviews and that researchers observed during site visits. 


Summary 

As the chapter shows, PACE’s intended program environment appears to be central to provid- 
ing gender-responsive programming. Analysis of staff interview data found that the program 
environment provided across the study centers was similar, with staff members focusing on 
safety, building positive relationships with the girls, recognizing girls’ strengths, and being 
aware of the girls’ prior exposure to trauma. The program environment provides the internal 
context for PACE’s academic and social services, which are detailed in Chapters 5 and 6. 


17 Center for Behavioral Health Research (2016). 
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Chapter 5 

Academic Services and Life Skills 


PACE’s academic services include academic advising, assessment, and middle school and high 
school classes. PACE also provides a gender-responsive life skills class called Spirited Girls!, as 
well as opportunities for volunteer service and career exploration. This chapter details how 
these services are provided at PACE centers and presents parents’ and girls’ perspectives of 
these services. The chapter concludes with data on the credits that girls earned while at PACE 
and findings on academic service receipt from the 12-month follow-up survey. Key findings are 
as follows: 


• Although the content of academic classes was similar to what girls would 
leam in regular public school, academics at PACE differed in student-teacher 
ratio, availability of academic advising, and a focus on individual, self-paced 
work. Classes were also provided in the context of a gender-responsive pro- 
gram environment, as described in Chapter 4. 

• Evidence was mixed as to whether the quality of instruction was similar to or 
better than what the girls received in public schools. PACE classrooms were 
rated similarly to public school classrooms on the CLASS-Secondary, a vali- 
dated tool for assessing the quality of classroom instruction. Although small 
class sizes offered an opportunity for teachers to work closely with girls, 
classroom observations found that many teachers offered limited academic 
support during independent work. Still, girls described the support they re- 
ceived from teachers as being superior to what they received in public 
school. 

• Spirited Girls!, PACE’s life skills curriculum, was offered at every center 
and was central to PACE’s gender-responsive programming. Spirited Girls! 
was the main vehicle for delivering career readiness activities and played an 
important role in providing opportunities for girls to volunteer in the com- 
munity. 

• On average, girls at PACE accrued credits at a rate that was on track with 
what is needed in Florida for on-time high school graduation. The program 
group was more likely to have been enrolled in school and to have received 
academic advising than the control group during the 12-month follow-up 
period. 
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• Staff members closely followed the PACE model for frequency of academic 
advising, assessment, and volunteer service. In areas where the model was 
less specific, such as the instructional approaches teachers should use and the 
types of career development activities centers should offer, there was more 
variation in the services girls received. 


Introduction to the PACE Academic Services Model 

The primary goal of academic services at PACE is to provide girls with the opportunity to earn 
credits and progress toward middle and high school completion. The PACE policy and proce- 
dures manual details what academic services each center should provide, including procedures 
for assessment, advising, course offerings, and instructional hours. The academic curriculum at 
PACE is driven by the Next Generation Sunshine State Standards, Florida’s modified version of 
the Common Core. All schools in Florida follow these standards; thus the content of the 
curriculum at PACE mirrors that of all public schools in Florida. 

Although the curriculum and academic services are similar across centers, how centers 
provide those services varies. These differences are primarily driven by the contracts that 
individual PACE centers have with their local school districts. These contracts specify how 
much the centers are reimbursed per girl served, what support the district provides to the center, 
and whether PACE has the authority to grant high school diplomas. Table 5.1 provides an 
overview of the key differences in academic services across the centers. 

The local school district also determines how girls will accrue credits while at PACE, 
which is one of the most significant differences between the centers in how academic services 
are structured. There are two models for earning credits while at PACE: programs based on seat 
time and, more commonly, competency-based programs. In competency-based education 
programs, girls earn credits by demonstrating mastery of the subject matter, typically through 
completing a packet that includes worksheets, quizzes, and tests for the subject following the 
Sunshine State Standards. In seat-time programs, girls earn credits much as they would in a 
traditional school: They must spend a certain amount of time in class and pass the class re- 
quirements. Five centers had seat-time programs, and one center used seat time for middle 
school girls only. PACE managers described pros and cons of each approach. Competency- 
based education programs, in use at nine centers, were viewed as advantageous to girls for 
catching up on credits: Girls who were motivated could finish classes faster than girls in seat- 
time programs. On the other hand, seat-time programs were seen as providing girls with a 
structure more similar to regular school and thus better preparing girls for their eventual 
transition out of PACE. 


50 



Table 5.1 


Academic Characteristics of PACE Centers 


Characteristic 

Centers 

System for accruing credits' 1 

Competency 

8 

Seat time 

5 

Combination of seat time and competency 

1 

Able to grant high school diploma 

6 

Offers GED preparation 

4 

Serves ESE students’ 1 

14 

Consultation-only services 

4 

Offers credit for Spirited Girls! classes 

13 

Academic staff turnover is a problem 

11 

Sample size 

14 


SOURCE: MDRC calculations based MDRC site visit data. 

NOTES: GED = General Educational Development certificate. 

“Competency programs award credits based on girls' mastery of 
the subject matter. Seat-time programs award credits based on girls' 
time in class and other class requirements. 

b PACE uses the Florida Department of Education definition of 
Exceptional Student Education (ESE), referring to programs for 
students with disabilities and gifted programs. Centers that provide 
consultation-only services are restricted to serving students who 
have low-level or periodic needs. 


Staffing of academic services was similar across centers. Teachers at PACE are certi- 
fied and often have experience in the public school system before they come to PACE. 
Paraprofessionals support teachers and assist with testing at some of the larger centers, but for 
the most part teachers are alone in their classrooms. An academic services manager or program 
director oversees the teachers and academic program. 

The average tenure at PACE for academic staff is shown in Table 5.2. Academic teach- 
ers (excluding those teaching Spirited Girls!) had an average tenure of less than three years. 
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Table 5.2 


Academic Staffing Characteristics at PACE Centers 


Characteristic 

Mean 

Sample Size d 

Class size 

11.0 

81 

Academic advising caseload 

17.1 

53 

Tenure of academic service manager (years) 

3.6 

14 

Teaching staff tenure at PACE (years) 

Spirited Girls! Teachers 

4.6 

9 

Other than Spirited Girls! Teachers 

2.9 

66 

Y ears of teaching experience 

8.9 

75 


SOURCE: MDRC calculations based on responses to the PACE 
evaluation staff survey. 

NOTE: a Sample consists of program staff members who completed 
the survey; sample size varies according to staff position. 


Interviews with management staff members at PACE revealed that staff turnover was greatest 
among teachers; 11 of 14 centers said that teacher turnover was an issue (Table 5.1). Staff 
members identified several reasons why they struggled to hire and retain teachers; competition 
with the school district was the primary reason. As a year-round program, PACE centers could 
not offer the same amount of time off to teachers as traditional schools. In addition, manage- 
ment staff members reported that girls at PACE could be more difficult to work with than the 
general student body in public schools, which also contributed to staff turnover. 


Classes at PACE 

When a girl enters PACE, her school transcripts and test scores are reviewed to determine her 
academic level and what credits she needs, and her academic schedule is created. Since girls 
enter the program at different points and need different credits, each girl at PACE will have a 
different schedule of classes. As a result, girls in the same classroom are often working on 
different subjects. For example, half of a social studies class may be working on world history 
while the other half is focusing on U.S. history. Due to rolling enrollment and the flexibility for 
girls to do self-paced work in competency-based programs, girls may be at different points in 
the curriculum even when they are working on the same subject in the same classroom. This 
means that the teachers must be able to support different levels and curricula within the same 
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class period. This type of “mixed” class was less common in seat-time programs, because seat- 
time classes needed a critical mass of girls to be working on the same subject at the same time 
— a scheduling challenge in itself, academic staff members reported. 

Teachers described a number of complications related to the mixed nature of the clas- 
ses. They noted that teaching multiple subjects at once requires familiarity with different 
curricula and each girl’s point of progress in the class. Mixed classes also limited the amount of 
full-group instruction that a teacher could provide during a class period. One tactic that teachers 
employed at many centers was to start the class with a “warm-up” lesson that involved the full 
class and was targeted to a subject that the whole class could benefit from before breaking up 
the class into smaller groups or independent work. Ultimately, the mixed nature of the classes 
meant that a large percentage of class time focused on independent work, with girls working on 
packets while the teachers provided individual support. 

PACE classes were much smaller than classes in the public schools that many girls at- 
tended before PACE. PACE classes are generally capped at 12 students, 1 while Florida class 
sizes for high school approach 20 students per class. 2 Teachers reported that classes usually had 
fewer than 12 students, which they unanimously said was a manageable size. In classroom 
observations during research visits, the average class size was just under seven. The smaller 
class sizes were due to student absences or girls being pulled from class for counseling or other 
activities. 

The PACE manual offers minimal guidance on the instructional methods teachers 
should use in their classes, simply stating that instruction should be individually tailored to the 
girl and teachers should accommodate different learning styles. In interviews, teachers said they 
made the primary decisions about the instructional approaches they used in class, and they 
reported using a wide variety of approaches. Classroom observation supported these reports. 
The research team observed 80 teachers across 14 centers during site visits, typically for the 
length of a full class period (50 minutes). 3 Researchers recorded the types of instructional 
methods teachers employed during the observations using four categories: teacher-led, group 
work, independent work, and other. During the observations, most teachers used two or more 
methods during the class period. Nearly two-thirds of classes included teacher-led instruction. 
About one-third included group work and about one-fifth included some other type of instruc- 
tion, such as watching a video or playing a game. The overwhelming majority of classes 


'PACE policy and procedures manual. 

2 Florida Department of Education (2014b). 

3 For each center, the research team attempted to observe each full-time teacher (excluding substitutes) for 
one full class period. Four to seven teachers were observed at each center. Three observations were subse- 
quently dropped from the analysis, because the usual classroom teacher was not present for the observation. 
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included individual work (84 percent), and 18 percent of class periods observed consisted of 
only independent work. 

Findings from the CLASS-S Observations 

During classroom observations, the team assessed interactions between teachers and 
students in the areas of emotional support, classroom organization, and instructional support 
using the Classroom Assessment Scoring System-Secondary (CLASS-S), a validated tool. Box 
5.1 describes the CLASS-S domains of emotional support, classroom organization, and instruc- 
tional support; Appendix B provides more details about the CLASS-S observations. During a 
50-minute session, researchers conducted two cycles of CLASS-S observations, with each cycle 
lasting approximately 25 minutes (15 minutes for observation and 10 minutes for coding). The 
researchers observed each teacher only once, rather than making repeat visits over a period of 
time, as is the procedure when CLASS-S is used for teacher performance evaluations; this limits 
what conclusions can be drawn about the quality of individual teachers, but the average scores 
across PACE provide an indication of the quality of PACE teachers overall. The distribution of 
scores by classroom is shown in Figure 5. 1 , 4 

Most PACE classrooms received midrange to high scores in the emotional support do- 
main, which reinforces the finding in Chapter 4 that PACE centers strive to create a positive 
enviromnent where staff members are sensitive to the needs of the girls. Most classrooms 
received scores in the mid- to high range in the positive climate and teacher sensitivity dimen- 
sions within this domain. However, PACE teachers scored lower on regard for adolescent 
perspectives, which reflects two findings: Girls were infrequently given leadership roles in the 
classroom, and teachers did not focus on connecting the content of the lessons to the girls’ lives. 
In an example of a class with high scores on regard for adolescent perspectives, the teacher 
involved the students in creating the rules for classroom behavior, offering them leadership 
opportunities and the chance to have input on the program. Classrooms with low scores tended 
to focus on independent work. 

Nearly all PACE classrooms scored in the mid- to high range on all classroom organi- 
zation dimensions, indicating that classrooms were orderly and productive, and behavior was 
generally not a problem. The high classroom organization scores are notable given that many 
girls have a history of behavioral issues when they come to PACE. Very few teachers received 


4 For each dimension, the lowest possible score is 1 and the highest possible score is 7. Low, midrange, and 
high scores were assigned to each calculated score using the ranges described by the CLASS-S developers: 
Scores of 3 or below were classified as low and scores of 5 or higher were classified as high. 
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Box 5.1 


Classroom Assessment Scoring System-Secondary (CLASS-S) Domains 

Emotional support focuses on how the teacher supports students’ social and emotional 
functioning in the classroom. This includes classroom climate, the relationship between the 
teacher and students, the responsiveness of teachers to student needs, and how the teacher 
creates opportunities for student autonomy and leadership. This domain comprises the follow- 
ing dimensions: 

• Positive climate 

• Teacher sensitivity 

• Regard for adolescent perspectives 

Classroom organization reflects the organization and management of students’ behavior, 
time, and attention in the classroom. This domain comprises the following dimensions: 

• Behavior management 

• Productivity 

• Negative climate 

Instructional support assesses the quality of instruction provided and the strategies that 
teachers use to develop understanding of content among students. Attention is paid to the 
format of the lesson, teaching methods, and feedback provided to students. This domain 
comprises the following dimensions: 

• Instructional learning formats 

• Content understanding 

• Analysis and inquiry 

• Quality of feedback 

• Instructional dialogue 

In addition, one dimension stands alone: 

• Student engagement 

SOURCE: Pianta, Hamre, and Mintz (2012). 
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Figure 5.1 

PACE CLASS-S Scores Across Classrooms, by Dimension 
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SOURCE: MDRC calculations based on observational assessments using the Classroom Assessment Scoring 
System (CLASS-S; Pianta, Hamre, and Mintz 2012). 

NOTES: For each dimension, the lowest possible score is 1 and the highest possible score is 7. Each 
calculated score was assigned to a category: Scores of 3 or less were classified low; scores greater than 3 but 
less than 5 were classified midrange; scores 5 or more were classified high. 

a Negative climate scores are displayed here in reverse: Higher scores mean a less negative climate. 
b Student engagement is considered a separate dimension outside the three domains. 
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low scores for behavior management or productivity; no teachers received low scores for 
negative climate . 5 However, though disruptive behavior was rare, off-task behavior by girls was 
commonly observed by the researchers and infrequently redirected by the teachers. Teachers 
noted during interviews that they would let girls who were having a bad day put their heads 
down or spend time outside class as a way of being sensitive to their needs, which may explain 
some of the allowances for off-task behavior. 

Most PACE classrooms scored in the mid- to low range in the instructional support 
domain, which is a common finding in other studies that have used the CLASS-S tool. The 
instructional support domain captures how teachers format lessons to engage students, the depth 
at which content is explored to support student learning and the development of higher-order 
thinking skills, and the quality of the feedback that students are provided on their work. PACE 
classrooms tended to score in the lower range in part due to the focus on independent work. The 
range of scores on the quality of feedback, with many classrooms scoring low and a few 
classrooms scoring high, reflect that teachers varied widely in their approach to supporting 
students during independent work. In classes with high quality of feedback scores, teachers 
consistently circulated among girls to provide support. They were aware of girls’ needs; 
sometimes, questions from a girl would prompt them to provide unplanned group instruction if 
it seemed like it would benefit others. In classes that scored low on quality of feedback, teachers 
gave responses that did little to advance girls’ understanding of the material, such as pointing 
them to a section of the textbook. There were also instances where teachers merely provided 
answers to the questions that the girls were struggling with, rather than helping the girls figure 
out the answer. 

Most PACE classrooms scored in the midrange for student engagement, reflecting that 
girls’ participation in classroom activities was mixed. A low score indicates that the majority of 
the class is distracted or disengaged, while a high score indicates that most students are actively 
participating in the classroom discussions and activities. Although some observed classrooms 
fell at each end of the range, most were in the midrange. Some girls would be on task, while 
others were off task. 

Analyses of the CLASS-S scores showed that they varied primarily across teachers 
within centers, with little variation in scores across centers. Further analyses (not shown) 
examined CLASS-S score differences by classroom or teacher characteristics, such as class 
size, instructional method, and teacher tenure, and found several statistically significant differ- 
ences. Classes that used only independent work had lower scores on all dimensions within 
emotional support, as well as lower scores in instructional learning fomiats, content understand- 

5 Negative climate is reverse coded; a low score on negative climate would indicate the presence of a nega- 
tive climate. 
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ing, and instructional dialogue. Longer teacher tenures were associated with higher scores in the 
classroom organization domain. Spirited Girls! classes scored higher on teacher sensitivity, 
regard for adolescent perspectives, and instructional dialogue, discussed further below. 

Several published studies have used the CLASS-S tool to evaluate the quality of in- 
struction in middle and high schools in the United States. 6 Each of these studies used a different 
methodology to select and observe classrooms. However, when comparing the CLASS-S scores 
across the studies, PACE classrooms perfonned similarly. Like PACE, teachers in the other 
studies tended to have midrange scores on emotional support dimensions, midrange to high 
scores on Classroom Organization dimensions, and midrange to low scores on instructional 
support dimensions. As in PACE, the lowest scores were observed in the analysis and inquiry 
dimension. 

PACE scores on the CLASS-S should be considered in the context of serving girls who 
were struggling in traditional schools, indicating that maintaining a productive learning envi- 
ronment may be more challenging, though PACE’s low staff-to-student ratio may help offset 
those challenges. Additionally, the reality of girls entering PACE at different levels and with 
different class needs means that PACE is not able to form classes in which every student is 
working on the same lesson at the same time, as typically seen in traditional schools, and 
instead must rely more on independent work. The instructional support scores reflect a very 
different style of classroom organization and teaching at PACE from what is common in 
traditional schools. 


Assessment and Advising 

The amount of academic advising girls receive at PACE sets the program apart from the 
traditional school environment. Florida public schools are not required to provide school 
guidance counselors. One study from 2012 found that Florida schools’ average ratio was one 
counselor for every 480 students. 7 At PACE the ratio was closer to one academic adviser per 12 
students. PACE requires that girls receive advising on their academic progress every other week 
from their assigned academic adviser. This adviser is different from the girl’s assigned social 
services counselor (discussed in the next chapter) and is usually a teacher, though two centers 
had a separate guidance counselor on staff who served as the academic adviser. Advising 
meetings cover academic progress (grades, progress toward earning credits, test results). 


6 Donaldson, LeChasseur, and Mayer (2016). CLASS results from MyTeachingPartner-Secondary Study 
and Measure of Effective Teaching Study appear in the CLASS-Secondary manual (Pianta, Hamre, and Mintz 

2012 ). ' ‘ 

7 Todd (2014). 
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PACE also requires each girl to have an Individual Academic Plan (LAP), which in- 
cludes goals in reading, math, writing, and career. The goals are often focused on skill gaps. 
PACE uses the STAR assessment to evaluate a girl’s reading and math needs, which then 
informs the IAP. 8 Teachers said the STAR assessment was useful because it gives specific 
information on a girl’s knowledge and needs and can help a teacher understand whether a girl’s 
prior grades reflect her actual knowledge and ability. The STAR assessment may be adminis- 
tered periodically to check on progress, and it is administered before girls make the transition 
out of PACE. 

According to the management information system data (shown in Table 5.3), the fre- 
quency of academic advising and assessment met PACE’s specifications. Among girls who 
enrolled in PACE, 93 percent took the STAR assessment and 96 percent had an IAP in place. 
Girls received academic advising on almost a biweekly basis, and advising sessions lasted, on 
average, about 1 3 minutes. Most girls who did not have an IAP or STAR assessment on file had 
short tenures (less than three months) at PACE. 


Life Skills Training 

As described in Chapter 4, key components of gender-responsive programming include provid- 
ing education about women’s health issues, promoting career opportunities for successful 
transitions to adulthood, and providing connections to the community. This section discusses 
PACE’s approach to providing gender-responsive programming through life skills training. 

Spirited Girls! 

Spirited Girls! (SG), PACE’s life skills class, is a central component of its gender- 
responsive programming. Box 5.2 provides an overview of the six developmental domains it 
covers. The study found that SG was taught for the most part by teachers, supervised by the 
academic manager, and included in a girl’s schedule alongside her other classes. The schedule 
for SG varied at each center, but at minimum classes would meet weekly. At some centers, SG 
classes met daily. The PACE model requires that girls take two semesters of the class, and at all 
but one of the centers girls could receive credit for the class. 

Until recently, PACE had used an SG curriculum developed in-house in the 1990s. 
Recognizing that the curriculum had become outdated, PACE recently began piloting the 


8 The STAR assessment is widely used in K-12 education throughout Florida and the United States. See 
Renaissance Learning Inc. (2016). 
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Table 5.3 


Participation and Satisfaction in Academic Services 

Measure Program Group 

Individual academic plan developed (%) 96.1 

Received academic advising (%) 93.5 

Number of sessions 17.0 

Number of sessions per month enrolled 1.8 

Time spent per session (minutes) 13.1 

STAR assessment administered 8 (%) 92.8 

Credits earned by length of stay b 

3 months or less 0.4 

More than 3 months to 6 months 1.7 

More than 6 months to 10 months 4.0 

More than 10 months to 15 months 7.3 

More than 15 months 10.1 

Participated in service learning projects (%) 73.6 

Number of projects participated in 6.8 

Number of projects participated in per quarter enrolled 1 .9 

Time spent on service projects (minutes) 88.5 

Among survey respondents c (%) 

Felt academic classes met needs 88.8 

Felt that PACE staff helped with job or career planning 82.1 

Felt Spirited Girls! taught her something new 78.9 

Sample size 613 


SOURCES: MDRC calculations based on data from the PACE management 
information system and responses to the 12-month follow-up survey. 

NOTES: Calculations include all program group members who enrolled at PACE, 
except as noted. 

a The STAR assessment (Renaissance Learning 2016) is widely used in K-12 
education. 

b This measure includes only girls who were no longer enrolled as of October 2016 
and refers to a girl's cumulative length of stay (N = 556). 

Percentages are based on responses to the 12-month follow-up survey among 
program group members randomly assigned through March 2015 (N = 376). 
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Box 5.2 

Spirited Girls! Curriculum: Content by Developmental Domain 

Physical. Healthy eating, drugs and alcohol, physical fitness. 

Sexual. Menstruation, sexual activity, sexual transmitted infections, contraception. 

Intellectual. Basic computer and test-taking skills, understanding grade point average and 
credit requirements, career exploration, job readiness. 

Emotional. Self-esteem, anger and stress management, coping skills, recognizing feelings. 

Relational. Healthy relationships, friendships, romantic relationships, family relationships. 

Spiritual. Topics are secular and aim to help girls think about their place in the world and their 
own approach to spirituality. 

SOURCE: PACE Spirited Girls! curriculum. 


evidence-based and gender-responsive Girls Circle curriculum, developed by the One Circle 
Foundation . 9 Researchers visited two centers that were using the Girls Circle curriculum. The 
remaining centers were using SG, and teachers explained that they would tailor the original SG 
curriculum to current topics, drawing on resources from outside the curriculum . 10 For example, 
the SG curriculum does not cover the topic of online bullying, so the teachers would bring in 
outside resources or news stories to cover that topic. 

SG classrooms were different from other academic classes. Unlike in many of the aca- 
demic classes, all girls in SG receive the same lesson at the same time . 11 This gave the teachers 
more opportunities to lead full-group lessons or engage the girls in group work. Indeed, SG 
classes had the highest frequency of teacher-led instruction and group work of all subjects 
observed by researchers. SG classes also showed statistically significant differences from other 
academic classes on the CLASS-S scores, scoring higher in teacher sensitivity, regard for 
adolescent perspectives, and instructional dialogue. These differences reflect the content- 
oriented discussions about real-life issues that were more prevalent in SG classrooms than in 
regular classrooms. 

Q One Circle Foundation (2012). The Girls Circle curriculum was first developed in 1994. 

10 At the time of publication of this report, all PACE centers were implementing Girls Circle as part of 
Spirited Girls! class. 

11 Since enrollment at PACE is rolling, not all girls in the SG classes would have been exposed to the same 
parts of the curriculum. 
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Career Exploration and Volunteer Service 

The PACE model includes a career component, which focuses on educating girls about 
possible career options and strengthening academic and soft skills to support work readiness. 
Upon entering PACE, girls take an assessment to identify career interests, and these become 
part of their IAP. The PACE model also requires each girl to participate in a volunteer service 
project each semester. SG was the primary vehicle to expose girls to career options and work 
readiness activities; activities in SG included resume writing, mock interviewing, and filling out 
job applications. Three centers had a separate career class apart from SG. 

Career development activities outside SG varied among the centers; though a few cen- 
ters provided robust career activities outside SG, such as internships or job shadowing, they did 
not seem to involve many girls. The ad hoc nature of such career development activities could 
be attributed to the lack of dedicated staffing and reliance on external partnerships to deliver the 
services. Also, since most girls return to middle or high school (or attend another alternative 
school) after PACE, careers may be a lower priority for centers than getting girls ready to go 
back to regular school. 

Girls’ participation in volunteer service projects is seen by PACE as promoting self- 
esteem, building work readiness skills, and contributing to the community. Most centers 
described providing volunteer service activities on a monthly basis. These were also typically 
delivered through the SG class or through another class or group. Activities included on-site 
activities, such as letter writing or putting together care packages for the homeless, and off-site 
activities, such as volunteering at nursing homes. Staff members said the off-site activities were 
more engaging for the girls but more challenging to implement for a variety of logistical and 
behavioral reasons. Some centers used participation in off-site community service projects as an 
incentive, so not all girls were eligible to participate in all projects. As shown in Table 5.3, 
about three-quarters of girls participated in projects, with a frequency of two projects per quarter 
enrolled. This level of participation in community service corresponded to PACE requirements. 


Girls’ and Parents’ Reflections on Academic Services 

Overall, girls and parents had positive experiences with academic services at PACE. As shown 
in Table 5.3, 89 percent of program group girls surveyed felt the academic classes met their 
needs. Eighty-two percent of girls felt that staff members were helpful with job or career 
planning, and most girls felt that they learned something new from SG. Box 5.3 presents one 
girl’s story about the benefits of PACE’s academic services and SG. 

In interviews, both while they were still in the program and after they had left, girls had 
mostly positive things to say about academic services. Most commonly, girls praised their 
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Box 5.3 

One Girl’s Perspective on Classes at PACE 


Brittany: Appreciated academic assistance and Spirited Girls! 

A guidance counselor at her previous high school referred Brittany* to PACE because she had 
fallen behind in credits. She was working toward her high school diploma and planned to 
graduate from PACE. 

Brittany emphasized the benefits of the engaged teachers and small class sizes she has experi- 
enced at PACE. “The classes are small . . . they don’t have a lot of students, so it’s better 
because they are able to focus on the students more, instead of public school where they have a 
whole lot of kids so they don’t have enough time to help you.” Brittany appreciated the way 
teachers interacted with the students: “The teachers are very helpful, and they do like to help 
the students, and they like their job.” 

Brittany also said she was more focused on her academic classes at PACE and therefore able 
to complete more work. She compared this with public school, where she was “busy listening 
to [her] friends,” which distracted her from her school work. The one drawback of PACE was 
that the classes offered were “basic,” meaning that PACE does not offer electives like foreign 
language or physical education as regular public schools do. 

Brittany also said that as a pregnant young woman, she benefited from the knowledge she 
gained in Spirited Girls! She enjoyed learning about her body, about health, and about how to 
care for herself and her baby. 

*Name has been changed. 


teachers and found them to be helpful. Some attributed teachers’ helpfulness to the small class 
size. One girl explained: “Because there weren’t a lot of students, you could have one-on-one 
time with the teacher and they can explain stuff to you more.” The small class also meant that 
there were fewer distractions. Some girls remarked that they felt the teachers were more 
invested in them, in line with the relational focus of the program enviromnent described in 
Chapter 4. One girl put it this way: “I guess I felt like they were more than just a teacher; they 
actually cared about you.” 

A minority of girls interviewed did not like the academic program. For some, the self- 
paced nature of the classes did not work for their learning style; they found the independent 
work boring and would have liked more teacher- led instruction. One girl said, “I just don’t like 
how I have to do the work on my own. When I do my work, I don’t really read it; I just skim 
and skim. I don’t leam nothing.” 


63 




Girls were also generally positive about the SG class. Among girls interviewed, four 
times as many girls said they liked SG as said they disliked it. Girls who liked SG mentioned 
enjoying the conversations in class, both the variety of topics covered and the relevance of the 
topics. One girl described it this way: “You leam about yourself as a girl. Your body. Some of 
the activities we do make you look at yourself differently, and your relationships differently. 
You get more in tune with yourself.” The minority of girls who did not like SG expressed 
discomfort with the topics discussed in SG and with group discussions about those topics. 

Girls interviewed described receiving a variety of career services from PACE. Services 
ranged from general career readiness activities, such as mock interviewing and career explora- 
tion, to more in-depth college and career planning services around specific career tracks. Girls 
also offered positive reflections on their participation in community service activities; they 
appreciated the opportunity to help others and thought the activities were fun. 

Parents and guardians interviewed had mostly favorable things to say about PACE’s 
academic services. In fact, most parents said that it was the academics that drew them to PACE 
in the first place. Like the girls, parents liked the small class sizes, which they viewed as 
providing more individual attention for the girls. Parents also liked the self-paced nature of the 
academic program and felt that it was a good opportunity for girls to catch up on credits. 


Academic Progress 

Catching up on credits was the primary academic goal for most girls at PACE. As described in 
Chapter 3, more than half the girls entering PACE had been held back in school at least once, 
and three-quarters had recently failed a class. PACE offers the opportunity for girls to earn those 
credits, and in centers with competency programs, staff members said that girls who were 
motivated could get back to grade level faster than they could in regular school. 

Table 5.3 shows the number of credits earned by girls who enrolled in PACE. 12 As ex- 
pected, the longer a girl was enrolled in PACE, the more credits she received. In Florida, 
students must earn 24 credits in high school to earn a standard diploma. 13 A yearlong course, 
such as Algebra I, would earn a student one credit. The PACE credit data suggest that girls on 
average earned credits at a rate similar to what they would in high school; students who were in 
PACE for more than six months but less than a year (the equivalent of one or two semesters) 
earned four credits. Students progressing normally in regular high school earn about six credits 
in two semesters. It is not possible to make direct comparisons between regular school sernes- 

12 This measure includes only girls who were no longer enrolled as of October 2016 and refers to a girl’s 
cumulative length of stay; 57 girls in the study were still enrolled as of October 2016. 

13 Florida Department of Education (2014a). 
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ters and credits earned at PACE by length of stay. However, considering that most girls in 
PACE were not accruing credits at the rate needed to graduate on time before they came to 
PACE, these data suggest that PACE helped girls catch up on their credits. The final report will 
include an analysis of school records data to estimate PACE’s impacts on school progress for 
study participants. 


Services Received 

Table 5.4 presents preliminary impacts on academic services received. These measures come 
from the 12-month follow-up survey and represent only a partial sample, because the 12-month 


Table 5.4 


One-Year Impacts on Academic Services Received 



Program Control Difference 


Outcome (%) 

Group 

Group 

(Impact) 

P- Value 

Ever enrolled in a school or educational program in past year 

99.1 

92.9 

6.2 *** 

0.000 

Received academic advising in past year 

81.1 

67.6 


0.000 

Primary source of academic advising a 





Parent, guardian, or other relative 

8.6 

44.1 

-35.5 


Someone from school 

8.5 

44.9 

-36.4 


Someone from PACE 

82.1 

2.2 

79.9 


Other 

0.8 

8.9 

-8.1 


Frequency of academic advising sessions 





More than once per month 

38.0 

33.8 

4.2 

0.293 

Once per month 

19.6 

11.5 

g 2 *** 

0.009 

1-3 times per year 

23.4 

22.4 

1.0 

0.770 

Never 

19.0 

32.4 

_13 4 *** 

0.000 

Sample size b (total = 668 ) 

407 

261 




SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 

NOTES: Results in this table are regression-adjusted, controlling for pre -random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a Responses include only sample members who reported receiving academic advising. Therefore, the 
measures are nonexperimental and statistical significance was not calculated. 

b Due to missing values, the number of girls included varies by outcome. The sample size reported 
here is based on responses to the 12 -month follow-lip survey among girls randomly assigned through 
March 2015. 
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follow-up period was not complete for the full sample at the time this report was written. 
Overall, findings are promising. Nearly all the girls in the study sample enrolled in a school or 
educational program during the follow-up period. The program group was statistically more 
likely to have been enrolled in school than the control group (99.1 percent versus 92.9 percent). 
The majority of girls in the control group enrolled in a traditional middle or high school, but 
many girls enrolled in other types of schools, including charter, alternative, and virtual schools, 
reflecting the wide array of academic options available in Florida. 

Girls in the program group were 13.5 percentage points more likely to have received 
advising. They received this service more frequently, and more often from a professional — 
either someone at PACE or another school. The control group was much more likely than the 
program group to report receiving academic advising from a family member. This finding 
reinforces the indications discussed above that PACE provides more academic advising than 
traditional schools, where many in the control group were served. 
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Chapter 6 

Social Services 


Through social services, PACE strives to address the nonacademic needs that may hinder a 
girl’s success in the program and beyond. Social services include case management and 
counseling, parental engagement, connections to supportive services, support for transitions, 
and follow-up contact. This chapter describes PACE’s social service provision, the experiences 
of girls and parents with those services, and the difference in social service receipt between 
program and control groups as reported in the 12-month follow-up survey. The key findings on 
social services are as follows: 

• PACE’s comprehensive social services set the program apart from most pub- 
lic schools. Girls in the program group were more likely than girls in the con- 
trol group to have received counseling and case management during the 12- 
month follow-up period. 

• PACE centers closely followed the prescribed PACE model for the frequen- 
cy of assessment, counseling, and parental engagement while girls were in 
the program. PACE does not specify the approach counselors should take 
with girls during their meetings, so the content of these sessions varied con- 
siderably. 

• Once girls left PACE, follow-up services were limited. Transition and fol- 
low-up services tended to be understaffed, which helps explain why imple- 
mentation of these services was more uneven. 

• Girls and parents reflected mostly positively on the social services they re- 
ceived while at PACE, describing the counselors as helpful and supportive. 

Girls were less satisfied with follow-up services. 


Social Services Structure and Staffing 

The goal of social services is to provide each girl with the support she needs to be successful in 
academics and build the foundation for success in the future. As one executive director ex- 
plained: “Social services give girls stability as they go about academics. They are able to focus 
on themselves for the first time at PACE.” Social services are where the differences between 
PACE and regular public school are most apparent — PACE offers far more support outside the 
classroom. As a point of comparison, one study found that the average ratio of social workers to 
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students in Florida was greater than 1 to 2,500.' As shown in Table 6.1, the ratio at PACE was 1 
to 17. 


Typically, a social service manager oversaw a team of three to five counselors in a few 
types of positions. Outreach counselors focused on recruitment and intake activities and had 
limited interaction with girls once they were enrolled in the program. 2 Day counselors had 
primary responsibility for girls enrolled in the program. Transition counselors typically became 
involved with a girl’s care as she neared the end of her time at PACE and were also responsible 
for follow-up services. Table 6.1 presents the characteristics of social service staff members. 
Counselors were at both the bachelor’s and master’s levels. Counselors had an average caseload 
size of about 17 girls. In the staff survey, about 10 percent of day counselors said that their 
caseload was not manageable; 30 percent of survey respondents indicated that their caseload 
was very or extremely manageable. Since counselors are not clinical staff, they are not required 
to have a license, and few reported that they were licensed. As Table 6.2 shows, a small number 
of centers had additional social services capacity, including staff therapists and nurses. Most 
centers also had interns on staff, and interns might carry a caseload at some centers. Unlike the 
situation with teachers, no managers reported that counselor turnover was an issue. 


Assessment and Care Planning 

PACE evaluates the needs of each girl enrolled through a comprehensive assessment process, 
which is documented in PACE’s management information system (MIS). The process starts 
with the Initial Needs Assessment (INA), described in Chapter 3, which helps determine 
whether the girl is a good fit for the program. The On-going Needs Assessment (ONA) is 
administered within the first 30 days of a girl’s enrollment in the program and provides a more 
detailed assessment of the girl’s history and her current needs. 

The ONA is organized into six domains: relational, intellectual, physical, sexual, emo- 
tional, and spiritual. The ONA is primarily administered through interviews between the girl 
and her assigned counselor, but the counselor can also involve other sources in completing the 
ONA, including parents, outside case managers or counselors, and a review of the girl’s files 
from previous schools or other providers. The ONA probes into sensitive areas, such as a girl’s 
sexual and abuse history. Counselors described tailoring the administration of the ONA to each 
girl: With some girls the ONA could be administered in one sitting; with others the counselors 


'Student Support Services Project (2012). 

2 Most centers added this position during the study period. Some centers eliminated this position after study 
enrollment concluded. 
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Table 6.1 


Characteristics of PACE Counselors 


Characteristic 

Value 

Sample Size 

Average tenure at PACE (years) 

3.7 

66 

Average years of experience 3 

7.3 

66 

Percentage with master's degree 

62.1 

66 

Percentage with bachelor's degree 

37.9 

66 

Percentage with license 13 

9.4 

53 

Average counselor caseload 1 

17.1 

53 

SOURCE: MDRC calculations based on responses 
staff survey. 

to the PACE evaluation 

NOTES: Values are based on responses to the PACE evaluation staff survey 
and include counselors, transition counselors, and outreach/intake counselors. 
a Measure includes experience during practicum and internships. 
b A license is not required for counseling positions at PACE. The sample 
size is smaller than for questions above because this question was added to 
the survey after staff members at three centers had completed it. 

c Sample includes only social service staff members who carry a caseload 
of girls for counseling. 

Table 6.2 



Social Service Staffing Characteristics at PACE Centers 

Characteristic 


Centers 

Full-time transition counselor on staff 


5 

Full-time therapist on staff 


4 

On-site nurse 3 


2 

Interns on counseling staff 

Interns carry a caseload 


13 

9 

Counseling staff turnover is a problem 


0 

Sample size 


14 


SOURCE: MDRC calculations based on data collected during site 
visits. 

NOTE: a Measure includes nurses provided by an outside partner. 
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would break it up over several meetings and build trust and rapport with the girl before asking 
the most sensitive questions. PACE requires that the physical domain of the ONA be completed 
on a girl’s first day in order to determine whether a girl has immediate medical, dental, or 
mental health needs. 

Counselors used the infonnation gathered in the ONA to create each girl’s care plan. 
Care plans are organized along the same domains as the ONA and focus on the specific goals 
that a girl will work on while at PACE. As shown in Box 6.1, care plans included up to six 
goals, aimed at both reducing risk factors and building strengths. Counselors used these goals to 
guide their sessions with each girl and to update parents on her progress during monthly 
meetings. Care plans were not directly used by noncounseling staff, but the counselors used the 
care plan and goals to inform discussions with other staff members about a girl’s care. 

Each girl’s progress is required to be reviewed at least monthly during a meeting of the 
care team, which includes the girl’s assigned counselor, academic adviser, and at least one 
manager. Typically, PACE centers held care review meetings every week or every other week. 
The researchers observed that these meetings were attended by most staff members — counse- 
lors, teachers, and managers. During these meetings, counselors discussed a girl’s progress 
toward her care plan goals and shared updates on her issues or needs, and teachers shared 
infonnation on her progress in class. Staff members discussed any steps that needed to be taken 
to help the girl reach her goals, such as addressing problems with attendance or behavior. For 
girls who were progressing well, the team would discuss ways to reward or acknowledge her 
perfonnance and keep her on track. Care review was the formal way PACE staff members 
shared infonnation about girls. Outside these meetings, staff members shared emergent issues 
about the girls through morning staff meetings, emails, and ad hoc meetings. 

The MIS data indicate that centers were closely following PACE requirements for ad- 
ministering assessments and creating and monitoring care plans. As shown in Table 6.3, 96 
percent of girls had their ONA administered within 30 days of enrollment, and 98 percent had 
their care plan developed within 30 days. Girls who did not have an ONA or a care plan tended 
to be those who left the program shortly after enrollment. 


Individual Counseling 

Based on the PACE policy and procedures manual, a girl should meet with her assigned 
counselor every two weeks to review her progress. According to MIS data (see Table 6.3), 
meetings between counselors and girls happened more frequently than the recommended twice 
per month. The average time per session was 39 minutes. Counselors reported that rather than 
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Box 6.1 


Example of Care Plan Goals 

Decrease substance abuse 

Increase anger management skills 

Decrease self-hami 

Increase positive self-esteem 

Decrease social isolation 

Increase healthy relationships 

SOURCE: PACE’s management information system. 


following a set schedule for the meetings, they would pull the girls from class at a time that was 
convenient for the counselor and the girl. For example, counselors would avoid pulling girls 
If om classes in which they were behind on their work. 

Although the primary purpose of these biweekly meetings is to review the girl’s pro- 
gress toward the care plan goals, the research team found that counselors used a broad range of 
approaches in the meetings. Most counselors described the care plan goals as a jumping-off 
point for discussions with the girls. Interviews with counselors revealed that there was a fine 
line between discussing goals with a girl and delving into sensitive topics that might require a 
more therapeutic approach. A quarter of counselors interviewed said they did not use clinical 
approaches in their work with girls, with some of those counselors stating that they were not 
qualified to provide clinical therapy. The majority of counselors, however, described bringing 
aspects of clinical practice into their session with the girls; the most common approaches 
mentioned were cognitive-behavioral therapy, solutions-focused therapy, and art therapy. 
Interview data suggest that the nature of topics girls discussed with their counselors often led 
counselors into playing a more therapeutic role. For example, a girl might have the goal of 
better handling her anger, which might stem from abuse by a family member. Helping the girl 
find a strategy for responding in upsetting circumstances could require addressing the underly- 
ing trauma or depression. 

Counselors used a variety of strategies in their work with the girls; most counselors said 
they tailored the approach to a girl’s individual needs and personality. For example, a counselor 
might use hands-on activities like worksheets or art projects with girls who were having trouble 
opening up to the counselor. A review of counselor case notes entered into PACE’s MIS system 
provides examples of the range of content these sessions could have. One type of session 
focused more on helping a girl support her academic progress or address behavior issues within 
the program. As an example of this type of session from the case notes: “[Counselor] reviewed 
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Table 6.3 


Counseling and Parental Involvement at PACE Centers 

Service Program Group 

Ongoing needs assessment (ONA) ever administered 2 (%) 94.0 

ONA administered within 30 days of enrollment 96.2 

ONA updated every 6 months’ 5 73.7 

Physical domain assessment completed at enrollment (%) 95.9 

Care plan ever developed (%) 96.9 

If yes, initial care plan developed within 30 days 97.8 

Girl reviewed at care review staff meeting (%) 99.5 

Initial care review staff meeting held within 1 week (%) 88.5 

Number of times reviewed at a care review staff meeting 12.5 

Girl received individual counseling 2 (%) 97.4 

Number of sessions 21.9 

Number of sessions per month enrolled 2.5 

Time per session (minutes) 38.7 

Girl attended psychoeducational or extracurricular group d (%) 85.2 

Number of group sessions 18.4 

Number of group sessions per month enrolled 1 .8 

Time per group session (minutes) 59.5 

Parent/guardian ever contacted 2 (%) 97.2 

Number of parent/guardian contacts attempted over course of enrollment 18.7 

Number of parent/guardian contacts attempted per month enrolled 2.5 

Sample size 613 


SOURCE: MDRC calculations based on data from the PACE management information system. 

NOTES: Calculations include all program group members who enrolled in PACE. 

“Measures are based on ONAs administered only during a girl's first enrollment. 
b Measure includes only girls who were enrolled for at least six months. 
c If a girl received more than one counseling session in a given day, only the most recently 
entered session of that day was accounted for in these calculations. 

d lf a girl attended more than one group session on a given day, only the most recently entered 
group session of that day was accounted for in these calculations. 

e If there was more than one parent/guardian contact on a given day, only the most recently 
entered instance of contact was accounted for in these calculations. Initial home visits could also 
be captured in these measures. 
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girl’s progress and addressed some concerns about girl’s academic performance because she is 
not showing as much progress as she could. She blames peers in the class for distracting her by 
their behavior. We discussed some options available to her in the classroom to separate herself 
and focus on her own academic success.” Another type of session included a focus on relational 
issues in the girl’s life, for example: “Girl reports feeling very anxious about a friend’s family 
situation. Girl reports that she is worried about this friend and cannot stop worrying about her. 
Discussed with girl the importance of maintaining personal awareness of how others’ issues 
impact her and the need to ‘fix’ other people’s issues.” Often, sessions would cover more than 
one topic. 

Girls’ Perspectives on Counseling 

Most girls interviewed had positive things to say about the counseling they received at 
PACE. They described talking with their counselor about their goals and the issues they faced at 
school and at home, and most girls said that their sessions with their counselor were helpful. 
Girls reported that the counselors were both problem-solvers who helped them with strategies to 
address the challenges in their lives and sources of emotional support, providing them with 
encouragement and reassurance. One girl said: “I feel like sometimes I’m going to break down. 
She believes in me. She tells me I can do it.” A minority of girls interviewed (less than 20 
percent) said that they did not open up to their counselors, either because they did not feel that 
they connected with the counselor or because they did not open up to anyone. Box 6.2 provides 
two different perspectives on counseling. 

Case Management and Ancillary Services 

Counselors also functioned as case managers, working with girls to identify unmet 
health or other needs and connect them with appropriate services, either through PACE or 
through partners. All PACE centers had informal partnerships with outside providers to connect 
girls to ancillary services. The most common referrals to outside services were for health 
services, such as reproductive or dental care, or for mental health services, when a girl needed 
services beyond what the center could provide. Counselors described several challenges with 
connecting girls with services outside PACE. One challenge was the availability of services in 
the community. Services available to the girls, such as free counseling, often had long waiting 
lists or were perceived to be of low quality. Another challenge with outside referrals involved 
the parents — referrals often required the parent to give approval or to schedule the appointment 
and arrange transportation. Lack of parental follow-through could mean that the girls were not 
able to receive the services they were referred to. 
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Box 6.2 


Two Perspectives on Counseling 

Kendyll: Counseling was helpful 

“It was okay to be me and to keep being who 1 am and not change for nobody.” 

This was among the key lessons that Kendyll* learned over the eight months she spent at a 
PACE center in central Florida. 

Kendyll began at PACE when she was 17. The center proved to be a safe, secure, and support- 
ive environment for her. She was able to make close connections to staff members as well as to 
some other girls at the center. 

In particular, Kendyll identified counseling as the most helpful component of PACE. She was 
struggling with depression and anxiety when she arrived, and though she continues to experi- 
ence those issues, she said that they were “not as bad” now, an improvement which she 
attributes to the work she did with her counselor. She and her counselor talked about how to 
handle herself when she became upset. Eler counselor also gave Kendyll small projects. For 
one of them, Kendyll wrote down her good qualities and kept the list in her binder so that she 
could easily remind herself of them whenever she got upset. Kendyll appreciated that her 
counselor and other PACE staff members took her depression seriously and made dealing with 
it a priority. She found it easy to talk to PACE staff members about her goals and problems 
because they would try to help her, rather than judge her. 

Roxane: PACE was not the right fit 

Although Roxane* spent about eight months at a PACE center in central Florida, she ultimate- 
ly did not find the program to be a good fit for her and does not believe that her experience at 
PACE benefited her. 

Roxane, who was 15 when she started at PACE, struggled most with how much she was 
expected to share her feelings there. She said that her counselor was nice and sometimes 
helpful, but she did not have the right type of attitude to respond to counseling because she did 
not want to talk about her feelings. Roxane was uncomfortable with discussing issues in her 
life or her goals with PACE staff members, though she said that she did talk with them about 
problems she had with friends and her grandfather, with whom she lives. Similarly, Roxane 
strongly disliked Spirited Girls! because the class also asked her to talk about her feelings. 

*Names have been changed. 


A few centers were able to provide health and mental health services on site. Four cen- 
ters had therapists on staff to work with girls who had clinical or more intensive needs. Two 
centers had a nurse on site, provided through a contract with an outside provider. These centers 
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reported that having these additional services on site not only lowered the barriers to services 
but also improved access to providers with experience serving the PACE population. 

Counselors also provided support to girls who needed it outside their biweekly sessions. 
Girls who were having emergent issues, such as a crisis at home, could request a same-day 
meeting with a counselor — either with their regular counselor, depending on availability, or a 
counselor-on-call. Counselors might also be called by a teacher to meet with a girl who was 
having behavior problems in the classroom. Counselors reported meeting daily, if needed, with 
girls who were in crisis. 


Group Counseling 

PACE centers provide psychoeducational or therapeutic groups to girls who need them. The 
PACE manual stipulates that participation in therapeutic groups is voluntary. Centers deter- 
mined which groups to offer based on the needs of the girls enrolled. Therapeutic group topics 
included anger management, grief and loss, and sexual abuse. Some centers offered nonthera- 
peutic groups that were more like extracurricular clubs offered in regular high school, such as 
gardening or book clubs. A few centers did not have any groups operating at the time of the 
implementation visits. Staffing for groups varied quite a bit — from counselors to interns to 
outside providers. 

Staff members described groups that generally met weekly for about an hour for 8 to 1 0 
weeks. Some centers had a specific day and time in their weekly schedule when girls would 
attend groups, with some girls attending therapeutic groups and other girls attending nonthera- 
peutic groups. Centers that had groups built into their weekly schedule required all girls to be 
enrolled in a group, though girls could choose a group that interested them. As shown in Table 
6.3, 85 percent of girls participated in groups, attending an average of 18 sessions. The data do 
not differentiate between therapeutic groups and nontherapeutic groups; thus the high participa- 
tion rate is likely to be related to the broad range of groups offered and the requirement at some 
centers that all girls participate. Interviews with girls supported the rates calculated using the 
MIS; 80 percent of girls reported participating in a group. Many girls said these groups were 
required, and their description of the groups emphasized extracurricular-type activities more 
than therapeutic ones. A few girls reported attending therapeutic groups at the recommendation 
of their counselors. 


Parental Engagement 

Parental engagement is a central component of the PACE model and gender-responsive 
programming more broadly. Parental engagement is built into the program model in several 
ways. First, counselors are required to conduct a home visit within 30 days of a girl starting the 
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program. The purpose of this visit is to assess the home environment, such as sleeping arrange- 
ments and the availability of food. Counselors then meet monthly, ideally face to face, with a 
parent to provide an update on the girl’s progress. Counselors must also make a home visit if a 
girl has three unexcused absences, and staff members check in with parents on an ad hoc basis if 
there are issues to be discussed, including positive updates on a girl’s progress. 

According to the MIS data (Table 6.3), counselors attempted to contact parents about 
two and a half times per month on average. Due to limitations in how the data on contacts were 
documented in PACE’s MIS system, it is not possible to determine how many of these contacts 
were successful. Staff surveys and interviews with counselors indicate that, in fact, many of the 
attempted contacts were not successful. The majority of staff survey respondents (84 percent) 
said that no more than half the parents were actively engaged in supporting their daughter’s 
goals. During site visits, counselors also described difficulties reaching and engaging parents, 
noting that parents’ phone numbers would change or they would not respond to phone calls. In 
addition, counselors said, it was difficult to schedule meetings with parents around their work 
schedules. Counselors reported that some parents were part of the daughter’s struggles, but the 
parents did not want to see or acknowledge their role. 

Some staff members described engagement strategies to improve parental involvement, 
such as helping meet the parents’ needs by connecting them to resources for food stamps or 
counseling. Another tactic was reaching out to parents when things were going well, not just 
when there was an issue, which was an element of PACE’s strengths-based approach. These 
positive reports could be a new experience for many parents accustomed to hearing from school 
only when their daughter was in trouble. Counselors described being flexible with meeting 
locations, such as going to a parent’s workplace to meet if the parent’s work schedule made it 
difficult to come to the center. Some centers tried to involve parents in special events or as 
volunteers at the center to encourage their connection to PACE. The PACE manual states that 
staff members should make “every attempt” to meet with parents face to face, but counselors 
provided varying descriptions of how far they would go to involve parents. About half de- 
scribed aggressive efforts to involve parents, such as going to their houses and workplaces to 
track them down. 

In interviews with the research team, parents described positive interactions with the 
staff at PACE. 3 Most parents said that the counselors did a good job of keeping them up to date 
on their girls’ progress and appreciated that the counselors would work around their schedules 
for meetings. Parents said that PACE initiated most of the interaction with the parents, and 


3 Parents who participated in the interviews cannot be considered a representative group of parents. Parents 
interviewed were those who answered the phone when the interviewer called to survey the girl and expressed 
interest in being interviewed by the researchers. 
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some were surprised when PACE would call them to check on a girl’s attendance or share 
something about her progress. The minority of parents who had less positive things to say about 
their interactions with PACE found the home visits intrusive or felt that the counselor should 
have shared more information, such as telling the parent what was discussed in counseling 
sessions. To illustrate some of the tensions counselors may experience in working with parents, 
one parent interviewed told of being upset with the counselor because the counselor reported 
her to the police for abusing her daughter. 


Transitions 

PACE is not intended to be a girl’s final step on an academic or career pathway. Most common- 
ly, the girl returns to her zoned school or enrolls in an alternative or charter school in the school 
district. There are two main limits on a girl’s stay at PACE. First, Florida Department of 
Juvenile Justice funding caps a girl’s time at PACE at 15 months, granting an extension in 
limited circumstances. Second, only six centers in the study were authorized by the local school 
district to grant high school diplomas; in the remaining centers girls had to enroll in a different 
school in order to graduate. Many girls made the transition in less than 1 5 months, depending on 
their academic progress and progress on care plan goals. Unplanned transitions — in which a 
girl drops out of the program or is asked to leave — were also common; two-thirds of transition 
staff members at centers said that unplanned transitions happened at least monthly. Table 6.4 
shows girls’ survey responses to why they left PACE without completing the program. Most 
commonly, girls gave reasons that indicated that the program was not a good fit for them, such 
as not liking the program, not getting along with other girls there, or being asked to leave by 
PACE. Some girls selected multiple reasons for leaving the program. 

PACE’s motto on transitions is “transition begins at intake”; planning for transition is a 
key part of the model. Girls are given a planned transition date when they start at PACE, and the 
eventual transition is supposed to be part of the care planning process and monthly meetings 
with parents. Interview data indicated that most planning for transition fell within 90 days of the 
transition date. Counselors had the primary role in planning for transitions for the girls on their 
caseload. 

Counselors described two main ways they supported the transition process. Most com- 
monly, they described working closely with the girl and her family to determine the best school 
placement, including helping the family with the enrollment paperwork. Counselors also 
worked to prepare the girl emotionally for leaving PACE. Some centers, often those with 
dedicated transition staffing, did this through a formal transitions group attended by girls 
nearing that point. Other counselors did this one on one by talking with the girl about her 
feelings about transition during regular biweekly meetings and working with her on coping 
strategies. 
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Table 6.4 


Transition Services at PACE Centers 


Service (%) 

Program Group 

Transition planning before exit 

Transition outcome checklist completed 

93.9 

Exit plan completed 

95.7 

Transition care plan developed" 1 

58.1 

Among survey respondents 

Girl felt transition planning was helpful after leaving PACE b 

59.2 

Reason for leaving PACE before completing the program 0 

Did not like the program 

27.8 

Did not like or get along with girls 

31.7 

Was asked to leave or transition 

22.2 

Did not like or get along with staff 

10.6 

Moved 

12.8 

Did not have transportation 

11.1 

Parent or guardian did not like the program 

9.4 

Expecting child or had child care problems 

6.7 

Had health problems or an injury 

6.1 

Incarcerated 

4.4 

Family member became ill 

1.1 

Other 11 

29.4 

Sample size 

578 


SOURCES: MDRC calculations based on data from the PACE management information 
system and the PACE evaluation 12-month follow-up survey. 

NOTES: Calculations include all program group members who have made the transition 
out of PACE, except as noted. 

“Calculations include only girls who were enrolled for at least 30 days and are based 
on a girl's first transition period (N = 523). PACE is not required to create a transition 
care plan or provide transition services to girls who were enrolled for fewer than 30 
days. 

b This measure is based on responses to the 12-month follow-up survey among 
respondents randomly assigned through March 2015 who attended PACE (N = 218). 

“These measures are based on responses to the 12-month follow-up survey among 
respondents randomly assigned through March 2015 who left PACE but did not 
graduate or complete the program (N = 180). Respondents could select more than one 
reason. 

d Examples of "other" responses include feeling the program was not helping, 
wanting to return to a prior school or go to a different school, having trouble with 
attendance, and finding academics at PACE inadequately rigorous. 
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In the final weeks before transition, the counselor is to complete the transition outcome 
checklist, which summarizes the girl’s progress while at PACE, and create an exit plan. The exit 
plan has infonnation about the girl’s next placement and is used to set academic and personal 
goals for her during the transition period. As shown in the top panel of Table 6.4, there was high 
compliance in completing the required paperwork before transition, with nearly all girls having 
complete transition paperwork. During interviews, counselors explained that in the case of 
unplanned transitions where there is not much warning, the counselor can complete the paper- 
work without the involvement of the family. The high compliance rates reflect such flexibility. 

Girls interviewed described varying transition experiences, which seemed to be associ- 
ated with their experience in the program and the terms of their transition. Girls who stuck with 
the program and had to leave because they reached their goals or the 15 -month limit described a 
coordinated transition process similar to that outlined in the manual. Girls who had unplanned 
transitions, such as having to leave because of behavior issues, said their transition process was 
not smooth and reported problems transferring credits accrued while at PACE. 

For the most part, parents interviewed had positive things to say about the transition 
process. They said that the staff provided them with information on the transition well in 
advance and helped with the logistics of the new placement. The parents who did not feel 
supported during the transition process were mostly parents of girls who had unplanned transi- 
tions, some of whom had been removed from the program by the parents. These parents 
described limited communication or support from the PACE staff. 


Follow-Up 

For girls who attended the program for at least 90 days, PACE requires 12 months of follow-up 
services after the transition. 4 The manual specifies that a designated PACE staff person makes 
contact monthly for the first three months and then quarterly. The purpose of these contacts is 
twofold: to document the girl’s situation (in school and in her living situation) and to connect 
girls with services if needed. As one counselor explained, “Just because they are leaving us 
doesn’t necessarily mean that we’re leaving them. They always have a safe place to go to.” 

The staffing structure for follow-up services varied by center. For 9 of 14 centers, fol- 
low-up was a part-time role, either staffed by a part-time person or part of the duties of a full- 
time staff member, such as a day counselor. At five centers, the transition counselor was a full- 
time role, and the counselor would typically become involved in a girl’s care during her last few 


4 At the beginning of the evaluation period, PACE required three years of follow-up services, but this was 
reduced to 12 months during the study enrollment period. Since MDRC’s visits to study implementation 
occurred after PACE had changed the policy, this report focuses on the 12-month follow-up period. 


79 



months at PACE to enable the girl to get to know the transition counselor before transition. 
Transition staff members who responded to the staff survey reported an average caseload of 35 
girls in follow-up; 29 percent said their caseload was not manageable. The academic staff 
played a minimal role in follow-up. Follow-up contacts between academic staff members and 
girls mainly occurred when girls reached out to teachers with whom they had been close. 

Staff members responsible for transitions described a few methods for keeping in con- 
tact with the girls, including phone calls, social media (such as Facebook), and in-person visits 
to the girl’s school. It was often difficult to stay in touch; respondents to the staff survey 
reported that they were able to reach fewer than half the girls on their caseload during follow- 
up. The level of effort varied. Five centers reported minimal efforts to contact girls, limited to 
calling the numbers on file, leaving voice mails, and taking girls off the call list after the 
required three attempts. The remaining centers described more intensive efforts that included 
going to girls’ homes or trying to use their social networks to get in touch with them. 

Many staff members indicated that the girls drove the level of follow-up: If a girl want- 
ed help from PACE she would reach out to the program. Girls who did not want support from 
the program essentially opted out of services by not responding to contact attempts. In the 
words of one girl: ‘“They call me but I ignore it.” A few staff members indicated that their 
personal relationships with the girls affected follow-up efforts: “Sometimes we try really hard to 
find them, sometimes if we can’t reach them we just let them go.” 

The PACE model requires that each girl in the follow-up period have a transition care 
plan in place, which the transition staff will review with the girls during the required contacts; 
similar to the first care plan, it contains academic, behavioral, and vocational goals. At most 
centers, however, the transition care plan did not seem to be a focus of follow-up services. Most 
transition staff members reported using it either not at all or in a limited way. The MIS data 
(Table 6.4) support this finding: Only 58 percent of girls had had a transition care plan. Centers 
varied in their completion rates. On the high end, five centers had transition care plans for at 
least 75 percent of girls, while four centers had rates of 25 percent compliance or below. 5 Most, 
but not all, of the centers with a full-time transition staff had higher compliance rates. A few 
centers without a full-time transition counselor also fell in the high range. This indicates that 
centers could have high compliance in this area with or without a full-time transition staff 
member. 

Because MIS data are not available on services provided to girls during the transition 
period, it is not possible to say what proportion of girls received services and the intensity of 


transition care plan compliance is calculated among girls in transition who had a length of stay longer 
than 30 days. 
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those services. Staff members described mainly connecting girls to resources in the community, 
though PACE centers offered a limited amount of services directly. The most common services 
provided to girls in follow-up were help connecting to academic services, help with job searches 
(such as providing interview clothes or resume writing), and counseling (either directly or 
through referrals to outside providers). 

Follow-up services were an area where implementation was more uneven across the 
centers, with lower rates of fidelity in quantity of services provided compared with other 
services. Although a few centers described robust services, most did the minimum — calling 
girls at the required intervals and connecting girls who sought help with supportive services. 
This finding is not surprising, given that PACE does not receive funding for follow-up services 
from the Florida Department of Juvenile Justice or the school district, so funds to support this 
work must be raised from other sources. As a result of the lack of funding, follow-up services 
lacked a dedicated staff person at most centers. Uneven implementation may also be reflected in 
the girls’ reported satisfaction with transition services — compared with the overwhelmingly 
positive responses on other measures, only 59 percent of girls surveyed reported that the 
transition plan was helpful after leaving PACE (Table 6.4). Girls interviewed also described 
mixed experiences with follow-up services. Of the girls who had made a transition, about half 
reported being in regular contact with PACE and receiving support. The remaining girls said 
they were not in contact with PACE; girls indicated that they program did not contact them or 
they were not interested in keeping in touch with the program. 


Impacts on Services Received 

Table 6.5 shows that girls in the program group were more likely to receive support for nonaca- 
demic needs than girls in the control group, according to the 12-month follow-up survey of 
girls. The program group reported much higher rates of receiving mental health counseling or 
therapy (64 percent compared with 46 percent) and having more frequent sessions. More girls in 
the program group also reported receiving help connecting to additional services (such as 
transportation, housing, or food) than in the control group (39 percent compared with 23 
percent). These impacts demonstrate that PACE offers girls greater access to counseling and 
case management than they would receive in a traditional school setting. 

Table 6.5 also shows differences in the receipt of services that are closely connected to 
the PACE model, such as attention to women’s reproductive issues and healthy relationships. 
The program group was much more likely to receive help around reproductive health (including 
access to contraception or testing for sexually transmitted infections). The program group was 
also more likely to have received help with social and emotional skills, such as dealing with 
peer pressure and developing self-confidence. 
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Table 6.5 


One- Year Impacts on Social Services Received 



Program Control Difference 


Outcome (%) 

Group 

Group 

(Impact) 

P- Value 

Received help finding services in the community 

38.5 

23.3 

15 2 *** 

0.000 

Primary source of help finding services in the community a 

Public, alternative, or virtual school 

8.1 

36.2 

-28.1 


PACE 

56.5 

6.9 

49.7 


Department of Children and Families 

7.3 

6.6 

0.7 


JPO or person from Department of Juvenile Justice 

or the court system 

8.2 

19.2 

-11.0 


Health or mental health center, clinic, or private practice 

7.9 

13.3 

-5.4 


Other 

12.1 

17.8 

-5.7 


Received mental health counseling or therapy 

64.2 

45.5 

lg 2 *** 

0.000 

Primary counseling or therapy provider a 

Health or mental health center, clinic, or private practice 

17.9 

47.3 

-29.5 


Public, alternative, or virtual school 

1.9 

12.7 

-10.8 


PACE 

65.1 

2.6 

62.5 


Department of Children and Families 

2.9 

5.0 

-2.1 


Department of Juvenile Justice, parole office, 

or the court system 

4.6 

5.8 

-1.2 


Community organization 

2.3 

7.4 

-5.1 


Other 

5.4 

19.2 

-13.8 


Frequency of counseling or therapy sessions 

Once per week or more 

39.3 

23.9 

15 4 *** 

0.000 

1-3 times per month 

19.9 

13.4 

6.5 ** 

0.043 

Less than once per month 

4.6 

8.0 

-3.5 * 

0.081 

Never 

36.2 

54.6 

-18.5 *** 

0.000 

Received help related to sexuality, sex, 

or sexual and reproductive health 

72.0 

57.8 

14 3 *** 

0.000 

Received help related to social and emotional skills 

80.3 

63.0 

12 3 *** 

0.000 

Sample size * 5 (total = 668 ) 

407 

261 




(continued) 
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Table 6.5 (continued) 


SOURCE: MDRC calculations based on responses to the PACE evaluation 12-month follow-up survey. 
NOTES: JPO = juvenile probation officer. 

Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

“Responses include only sample members who reported receiving the services. Therefore, the measures 
are nonexperimental and statistical significance was not calculated. 

b Due to missing values, the number of girls included varies by outcome. The sample size reported here is 
based on responses to the 12-month follow-up survey among girls randomly assigned through March 2015. 
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Chapter 7 

Conclusion 


This report, along with the companion brief on the principles and evidence behind gender- 
responsive services that uses PACE as a case study, adds to the body of knowledge about how 
gender-responsive programs operate in actuality. Specifically, PACE creates a culture that is 
sensitive to the needs of girls as a framework for providing its services: PACE centers provide a 
safe, relationship-focused environment, and program staff members emphasize fostering girls’ 
individual strengths while understanding how the effects of trauma may influence the girls’ 
responses and behavior. This culture is infused into all aspects of program delivery, as PACE 
offers a combination of services that is hypothesized to meet the specific needs of girls at risk 
for poor life outcomes. Program components such as comprehensive assessments, life skills 
classes, and parental engagement — along with the program enviromnent — align with princi- 
ples of gender-responsive programming. 

The report also highlights the ways that PACE is different from the other services avail- 
able to girls in the communities where PACE operates. Few programs offer the same combina- 
tion of services, and those that do provide a similar array of services to both girls and boys. In 
most cases, girls in the control group attended a more traditional public school in their commu- 
nity. Girls attending PACE were more likely to participate in counseling sessions and receive 
academic advising — and with greater frequency — than girls in other academic settings. 
PACE’s staff-to-girl ratio provides for more individual attention and support in both academic 
and social services. 

This evaluation also examines replication of the program — often a difficult task, and 
one of great interest to policymakers, practitioners, and researchers. PACE was able to imple- 
ment its model consistently in a diversity of locations — not only providing the same services 
but establishing a similar program culture in centers across the state of Florida. As described in 
this report, PACE successfully specifies its model through principles shared with the centers 
and a lengthy policy and procedures manual, bolstered by comprehensive staff training and the 
use of program data to track implementation. The central office plays a key role, monitoring 
fidelity and providing guidance or support as needed. This combination of methods offers 
lessons for the field. 

PACE centers did vary in some ways. Variation tended to occur in areas where the pro- 
gram model did not offer specific guidance: for example, the approaches used by counselors in 
their sessions with girls. This is another important lesson, underscoring the trade-offs programs 
face when thinking about how — and how thoroughly — to define their model and monitor 
fidelity. Programs must decide how much discretion staff members have in providing services: 
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Research on adaptation and fidelity suggests that in some cases adaptation is beneficial, as it 
allows staff members to tailor an intervention to the needs of the population served. 1 Finding the 
balance between specification and flexibility is an ongoing discussion for programs such as 
PACE. 


PACE continues to assess the services provided and its implementation practices. Over 
the course of the evaluation period, PACE implemented a new management structure at its 
central office and revised policies on program eligibility, among other changes. On the subject 
of eligibility, the management team at the state office collaborated with leaders at the centers to 
draw on their hands-on knowledge of current practices. A revamping of the life skills curricu- 
lum is under way as well. Working groups have been formed to delve into findings shared from 
the implementation research, such as the content of counseling sessions and the instructional 
methods used in the classrooms. In addition, PACE continues to evaluate whom it should serve 
and how the larger societal conversation about gender identity plays out at the centers. 

The organization also must continue to examine its place in the ever-changing academic 
landscape in Florida. Some communities offer a wide array of academic options, and PACE 
seeks to differentiate itself and the services it offers. As a whole, the state offers an increasing 
number of charter, alternative, and virtual (online) school options. In the 2013-2014 school year, 
more than a quarter of kindergarten through twelfth-grade public school students chose a school 
other than the one they were assigned. 2 Florida offers more virtual school options than any other 
state; online learning is available in all its school districts. 3 In addition, Florida has experienced 
an increasing enrollment in charter schools. From the 2010-2011 school year to 2015-2016, 
over 130 charter schools were created, serving an additional 90,000 students from prekindergar- 
ten to twelfth grade. 4 Finally, there has been a notable increase in single-gender education 
options within the state, including in the communities served by PACE. 5 

Attitudes about approaches to delinquency continue to evolve as weft, both within Flor- 
ida and nationwide. In schools, there has been a move away from “zero tolerance” discipline 
policies that resulted in expulsions and a move toward handling behavior problems within the 
school setting. 6 This probably means fewer referrals for girls with behavioral issues to programs 
like PACE. Or, perhaps, girls might now be referred further down the line, when their behavior- 
al problems have grown to be too much for a school to handle. 


'Dusenbury, Brannigan, Falco, and Hansen (2003). 
2 Florida Department of Education (2015). 

’Florida Department of Education (2017). 

4 Florida Department of Education (2016). 
’Thompson (2015). 

6 Alvarez (2013). 
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Similarly, practices in the juvenile justice system are shifting. Recently, there has been a 
nationwide movement to confine fewer delinquent youth. 7 In Florida, 38,267 young people 
(ages 10 to 17) were arrested in fiscal year 2015-2016, down 34 percent from 57,597 in fiscal 
year 2011-2012. Females are making up a smaller percentage of juvenile arrests, falling from 30 
percent to 28 percent in the same period. 8 

The implementation findings herein add to the evidence base on gender-responsive 
programming and its effectiveness. The companion brief delves further into the history and 
literature of the gender-responsive approach, using PACE implementation as a case study. 9 In 
2018, a final report will present the results of the impact study, further reflections from girls on 
their experiences, and a cost-effectiveness analysis. The impact findings, which will provide a 
rigorous assessment of PACE’s effectiveness as a gender-responsive program for girls, will 
look at such outcomes as academic engagement and progress, involvement in the juvenile 
justice system, healthy relationships, and risky behavior (such as high-risk sexual activity and 
substance abuse). The cost-effectiveness analysis will evaluate the costs of PACE in the context 
of its outcomes for girls. Together these reports will provide needed evidence on a gender- 
responsive approach to helping girls. 


7 Data from Kids Count Data Center (2013): In 2006, 289 per 100,000 young people resided in a juvenile 
detention center, correctional facility, or residential facility. By 2013, the rate had dropped to 173 per 100,000 
young people. 

Florida Department of Juvenile Justice (2016). 

9 Treskon and Bright (2017). 
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Appendix A 


Supplementary Tables on Sample Member 
Characteristics and Risk Factors 




Appendix Table A.l 


Characteristics of Sample Members at Baseline, by Random Assignment Group 


Characteristic 

Program 

Group 

Control 

Group 

Age (years) 

14.6 

14.8 

Age (%) 



11-12 

8.8 

7.9 

13-14 

33.7 

30.8 

15-16 

47.9 

51.9 

17 or older 

9.6 

9.5 

Race/ethnicity (%) 



Black, non-Hispanic 

44.6 

45.9 

Hispanic 3 

15.8 

16.3 

White, non-Hispanic 

38.9 

36.9 

Other 

0.7 

0.9 

School level at referral to PACE (%) 



6th grade b 

9.8 

7.2 

7th-8th grade 

35.3 

40.1 

9th- 10th grade 

45.9 

44.4 

1 1th- 12th grade 

9.0 

8.2 

English is second language (%) 

2.1 

2.1 

Qualifies for special education or ESE C (%) 

11.0 

11.7 

People participant lives with (%) 



Two parents 

34.9 

34.7 

Single parent 

52.9 

50.2 

Relative 

9.8 

11.7 

Other d 

2.4 

3.3 

Family income e (%) 



$28,050 or lower 

43.0 

38.5 

$28,051-$44,900 

35.2 

35.8 

Above $44,900 

21.8 

25.7 

Family has had case with the 



Florida Department of Children and Families (%) 

43.0 

41.0 

Sample size 

679 

455 

(continued) 
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Appendix Table A.l (continued) 


SOURCE: MDRC calculations based on data from the PACE management information system. 

NOTES: No statistically significant differences were found between the program and control groups on 
any characteristics. 

a Sample members are coded as Hispanic if they answered "yes" to Hispanic ethnicity. 
b This category includes sample members who were in fifth grade at the time of referral. 
c PACE uses the Florida Department of Education definition of Exceptional Student Education (ESE), 
referring to programs for students with disabilities and gifted programs. 
d " Other" includes nonrelative or foster care. 

e The U.S. Department of Housing and Urban Development's guidelines were used in the data 
collection process to determine which income range the participant's family fell into. Since these figures 
could vary by county and household size, the ranges presented here correspond to the statewide income 
limits for low income and very low income for a four-person household in Florida in fiscal year 2014. 
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Appendix Table A.2 


Risk Factors of Sample Members at Baseline, 
by Random Assignment Group 


Characteristic (%) 

Program 

Group 

Control 

Group 

School engagement 



Recently expelled or suspended 11 

39.5 

39.8 

Currently enrolled in school 

72.6 

73.8 

Skipped school at least 3 times in past 2 months 

33.0 

37.5 

Had more than 1 5 absences in past 3 months 

42.9 

39.7 

Held back at least once 

51.2 

52.8 

Failed 1 or more classes in past 6 months 

76.3 

77.2 

Has a learning disability 

30.5 

28.1 

Attention deficit disorder 

20.9 

17.2 

Dyslexia 

1.3 

1.7 

Other learning disability 

8.7 

9.1 

Delinauencv 



Ever arrested 15 

28.3 

26.9 

Ever arrested for domestic violence 

4.8 

7.8 * 

Ever charged with burglary 

4.4 

4.0 

Ever charged with a drug crime 

4.2 

2.4 

Ever stolen from family, home, or neighbors 

16.9 

16.4 

Ever been on probation 

12.3 

13.0 

Currently on probation 

10.0 

10.3 

Has family member with criminal history^ 

63.8 

64.5 

Has friends with delinquent record or who engage in 



delinquent behavior 

48.9 

51.2 


(continued) 
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Appendix Table A.2 (continued) 


Characteristic (%) 

Program 

Group 

Control 

Group 

Health and safety 



Currently using tobacco d 

10.6 

8.8 

Currently using drugs or alcohol 6 

15.0 

14.5 

Ever sexually active 

44.1 

44.1 

Currently pregnant 

1.8 

0.7 * 

Ever run away from home 

27.9 

27.3 

Ever had thoughts about harming/killing herself 

38.6 

40.5 

Abused/neglected* 

38.4 

37.6 

Neglected 

9.4 

8.3 

Physically abused 

14.9 

17.3 

Sexually abused 

15.0 

15.3 

Emotionally abused 

21.3 

22.2 

Sample size 

679 

455 


SOURCE: MDRC calculations based on data from the PACE management information system. 

NOTES: Certain characteristics listed here were captured in two different ways during the 
random assignment period, as noted below. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

a For approximately half of the sample, this was defined as being currently expelled or 
suspended. For the other half of the sample, this referred to one or more expulsions or 
suspensions in most recent school term. 

b In the juvenile justice system, people are not technically "arrested"; the terminology used is 
either "incurred a charge" or "referred." 

c For approximately half of the sample, this measure referred to a criminal record (including 
imprisonment, probation, parole, and house arrest) for a parent, guardian, or sibling of the 
sample member. For the other half of the sample, "family" included other members of the 
household as well. 

d For approximately half of the sample, this was defined as having used tobacco three or more 
times in past 30 days, and for the other half of the sample this was defined as currently using 
tobacco. 

e For approximately half of the sample, this was defined as having used drugs or alcohol three 
or more times in past 30 days, and for the other half of the sample this was defined as current 
drug and or alcohol use. 

f For approximately half of the sample, this measure referred only to documented instances of 
abuse or neglect. For the other half of the sample, the measure also included suspected incidents 
of abuse. 
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Appendix B 


CLASS-Secondary Observations 




As part of the implementation data collection on PACE’s academic services, the research team 
used the Classroom Assessment Scoring System-Secondary (CLASS-S) observation tool. The 
CLASS-S is a validated measure for assessing interactions between teachers and students in a 
secondary classroom setting. 1 


The CLASS-Secondary System 


Observers using the CLASS-S score the classroom on 12 dimensions. These dimensions are 
organized under three domains, except for student engagement, which stands alone. 


Emotional support 

Classroom 

organization 

Instructional support 


• Positive climate 

• Teacher sensitivity 

• Regard for 
adolescent 
perspectives 

• Behavioral 
management 

• Productivity 

• Negative climate 

• Instructional learning 
formats 

• Content understanding 

• Analysis and inquiry 

• Quality of feedback 

• Instructional dialogue 

• Student 
engagement 


Observers give a score ranging from 1 to 7, with 1 being the lowest and 7 the highest. 
Individual scores must be whole numbers. CLASS-S coders generally observe for a cycle, take 
time to note the scores, and then repeat the process for another observation cycle. 


Training in CLASS-Secondary 

A trainer from Teachstone, the company that oversees the CLASS-S, led a two-day training for 
PACE evaluation team members in September 2014. The training materials included a manual, 
scoring sheets, and an extensive library of videos, hollowing the training, team members 
completed and passed a reliability test to receive formal certification. Team members who 
visited centers more than six months after training was complete were required to take and pass 
a calibration test (provided through Teachstone) before the visits. 


Observations at PACE Centers 

During implementation research visits to the 14 participating centers, all teachers (except 
substitutes) who were present on the day or days of the visit were observed in their academic 
classrooms, including the life skills instructor. The researchers conducted a total of 80 observa- 

University of Virginia, Curry School of Education (2015); Piante, Hamre, and Mintz (2012). 
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tions between November 2014 and March 20 15. 2 No interviews with teachers were scheduled 
before observations of their classes to avoid causing bias on the part of the observer. 

The goal at each center was to observe each teacher for one class period. This allowed 
for two cycles of CLASS-S observations, with each cycle involving one 10- to 15-minute 
period of observation, followed by a period of scoring. Observers referred to their CLASS-S 
manuals during the scoring process for guidance. Observations were not conducted if the class 
spent the entire period in testing or watching a video. No specific minimum class size was 
required for the observation to take place, as long as the number of students present was not 
atypical. In a handful of cases, classrooms with very few students (one to three) were observed. 

Observers occasionally encountered challenges to scoring classrooms on site, such as 
teachers who engaged them in the lesson or disruptions from students entering and leaving the 
classroom during the observation cycle. Questions and concerns were discussed regularly during 
team meetings and sometimes in consultation with Teachstone, which provided useful guidance. 


Inter-Rater Reliability 

To determine whether classroom visitors were scoring in a consistent manner, about 25 percent 
of classes observed were “double-scored,” meaning that they were observed and scored by two 
site visitors during the same period. Inter-rater reliability (IRR) was established by comparing 
the scores of the two observers. For CLASS-S, the two scores on each dimension had to be the 
same or within one point of each other to be considered in agreement. Overall, the IRR was 88 
percent, comfortably above the standard threshold of 80 percent. In comparisons on specific 
dimensions, IRR scores ranged from 79 percent to 97 percent. 


Calculating CLASS-S Scores 

CLASS-S scores were calculated for each dimension by averaging the scores from the two 
observation cycles in a specific classroom. For classrooms that were double-coded, the class- 
room score was calculated by taking the average of the two observers’ average scores. To create 
the overall score for a dimension at each center, the scores for all classrooms on that dimension 
were averaged. The negative climate score was reverse coded so that higher scores mean less 
negative climate. Scores of 1 to 3 are classified as low, scores above 3 but below 5 are classified 
as midrange, and scores of 5 to 7 are classified as high. 


2 Three observations were subsequently dropped from the analysis because the usual classroom teacher 
was not present for the observation. In two cases, a substitute teacher was present; in the other, the academic 
services manager had stepped in. 
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Appendix C 


Data Collection and Analysis 




Random Assignment Procedures 

Each of the 14 PACE centers used similar procedures to conduct random assignment from 
August 2013 through October 2015. Specifically, each center used its existing intake and 
eligibility procedures to determine whether an applicant was a good fit for PACE. During intake 
meetings, trained PACE staff members used MDRC -provided tools, including a short video, to 
discuss the evaluation and gather consent from the girl and her parent or guardian. 

Once a girl was determined eligible for PACE and her study paperwork was complete, 
PACE staff members entered a few pieces of her infonnation into MDRC’s web-based random 
assignment system. The system was set up to assign 60 percent of girls to the program group 
and 40 percent to the control group. This random assignment ratio was consistently used across 
sites. Random assignment was usually conducted the same day a girl was deemed eligible for 
PACE, and she was notified of the random assigmnent result within 24 hours. 


PACE’s Management Information System 

MDRC processed two types of data from the PACE management infonnation system (MIS): 
data collected at intake on all sample members’ demographics and risk factors, and program 
participation data collected once a sample member enrolled at PACE. 

Every two months from August 2013 through December 2015, MDRC pulled the in- 
take data directly from the PACE MIS. Though each data pull was cumulative, any demograph- 
ic or risk factor infonnation that had been updated from a prior pull was not used. Information 
on program enrollment and transition dates, also included in the intake data, was updated with 
the processing of each pull. In order to capture the most recently updated infonnation on 
program entry and exit for this report, MDRC made a few additional pulls of these data in 2016, 
with the last file processed in October 2016. 

Because retrieving program participation data from the MIS was more complex, the 
PACE administrator perfonned this task. From August 2014 to November 2016, cumulative 
program participation data records were periodically pulled for study participants only. MDRC 
worked extensively with the PACE administrator to understand which elements from the 
Efforts-to-Outcomes (ETO) software conesponded to the program participation measures of 
interest. Each data pull produced about 20 separate extracts, with each extract providing data on 
a specific type of program activity. MDRC then processed each data pull, which included 
perfonning checks for duplicates and invalid data and restructuring the data as needed to 
produce final measures of program participation across several domains. If any study partici- 
pants were missing from an extract, MDRC notified the PACE administrator, who was often 
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able to recover the missing data. Most of the program participation data recorded in ETO is 
documentation of an activity that occurred. In most cases, if a girl did not have any data entered 
for a specific activity, MDRC assumed no participation. 


Twelve-Month Follow-Up Survey of Girls 

A follow-up survey was administered to sample members by phone or in person approximately 
one year after random assignment. Fielding of the survey began in October 2014 and ended in 
December 2016. The survey covered a number of domains, including program satisfaction, 
academic progress and employment, outlook and positive youth development, social support 
and mental health, risky behavior, and control group service receipt. Survey responses presented 
throughout this report are from participants enrolled in the study between August 2013 and 
March 2015, about two-thirds of the full study sample. While all sample members were ap- 
proached to complete the survey, not all girls could be reached or agreed to participate. Specifi- 
cally, the response rate for this period was 7 1 percent overall; 407 girls in the program group 
(73 percent) and 261 in the control group (69 percent) responded to the survey. 


Staff Survey 

The staff survey was fielded to 299 PACE staff members at 16 centers using a list of email 
addresses obtained from PACE headquarters. The list included all staff members employed at 
the centers, including part-time administrative support staff. The survey was piloted in Fall 2014 
to the staff at three centers, and a revised version of the survey was fielded to the remaining 13 
centers in early 2015. Two centers receiving the survey ultimately did not participate in the 
implementation study because of an inadequate number of girls in their research samples. The 
survey results for these two centers are not included in the results presented in this report. The 
staff survey had a response rate of 9 1 percent. 


Missing Data in Quantitative Data Processing 

Program staff members enter the baseline data as part of their existing intake procedures. This 
information is required for a program eligibility determination before random assignment. 
Values were imputed for any missing baseline data used in the service receipt impact analysis. 

Sample members with missing data on any of the survey responses or outcomes were 
removed from the impact analysis for that particular measure. In the case of the 105-item 
Organizational Social Context measure administered in the staff survey, cases were eliminated 
if they had 1 1 or more items missing, were inconsistent in responses, or showed irregularity of 
response patterns. In all, eight cases were eliminated. 
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Qualitative Data Analysis 

Qualitative data were primarily gathered from interviews with staff members, girls, and parents 
using structured interview and observation protocols. A team of 12 researchers participated in 
the site visits. After the visits, the researchers recorded the information gathered in structured 
write-up templates designed to ensure that similar data were collected across staff roles and 
centers. A team of two implementation leads reviewed each write-up for clarity and consistency. 

All qualitative data were uploaded and coded in Dedoose, a mixed-methods analysis 
software. Data were coded by a small team of four researchers using a prespecified set of codes. 
All coded data used in this report were checked for proper code application by the lead imple- 
mentation researcher. Qualitative data were also attached to both a staff- or individual-level 
descriptor set and a center-level descriptor set in Dedoose. Descriptor sets are categorical or 
numeric variables that can be used to create subgroups of centers or staff members within 
Dedoose to aid in the analysis of the qualitative data. The descriptor sets were used to analyze 
the coded data across staff roles and centers to assess variation. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to leam not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 



